Explains how multimodal models turn images into many tokens, affecting prefill performance, unchanged routing, and benefiting from cached image encodings to reduce compute.
Requests are changing: users increasingly send images—screenshots, receipts, photos, or video frames—alongside text. Models that accept both text and images are called multimodal. The good news: most of the architecture and runtime principles you already know still apply. Only a few behaviors shift, and those shifts have clear operational consequences.
What changes and what stays the same
The transformer-based core remains: text and image-derived representations are processed in one unified token stream.
The practical differences come from how images are encoded, how much work that encoding adds to the prefill step, and the potential to cache expensive image encodings.
At a high level, three operational consequences follow directly from how multimodal models are implemented:
#
Consequence
Operational impact
1
Images become many tokens
One image can be as expensive as many pages of text; prefill work increases.
2
Routing and scheduling are unchanged
Endpoint pickers, scorers, and pools operate on the same sequence-level metrics.
3
Reuse via cache
Encoded image prefixes can be cached and reused across turns, reducing cost.
Consequence 1 — images → encoder → tokens (prefill grows)
Multimodal models do not embed a separate “brain” for images. Instead, an encoder converts an image into many numeric representations—either discrete tokens or continuous embeddings. Depending on resolution and model design, an image can become hundreds (or more) of entries in the token stream.
Image data is first encoded into numeric tokens or embeddings. After encoding, the transformer consumes a single unified sequence; some sequence entries originate from pixels rather than words.
Those image-derived representations splice into the same sequence as the surrounding words. From the transformer’s perspective there’s no separate object for images—just a stream of representations. That makes the usual token-stream processing rules apply directly.Because a single image can expand into many tokens, multimodal requests frequently skew toward a large prefill (reading the context + encoded images) but a small decode (the desired textual response). That means time-to-first-token (TTFT) and prefill performance become the dominant UX metrics. Splitting and optimizing prefill vs decode paths (scheduling, specialized pools, hardware tuned for large reads) yields significant efficiency gains.
The phases (prefill vs decode) didn’t change — only their relative proportions did.Consequence 2 — scheduling/routing treat image tokens like any other tokens
You do not need fundamentally new routing or scheduling concepts for multimodal requests. Endpoint pickers, scorers, queue-depth metrics, pool selection and load profilers operate on sequence-level signals. Image-derived tokens occupy the same queue slots, affect the same gauges, and are evaluated by the same scorers and profilers as word tokens.In practical terms: the scorer scores, the profiler schedules, and the pool binds—unchanged.
Even the front door often looks the same. Many chat-style APIs accept images in the same message payload as text. A common pattern is embedding image entries alongside text in a message content array. For example:
Consequence 3 — expensive encodings can be cached (KV cache)
Here is the big efficiency win: once an image has been encoded into tokens or key/value blocks, those representations can be reused. The same contract scanned over multiple turns, the same dashboard screenshot an agent consults repeatedly, or a product photo referenced in many tickets—these become repeated prefixes of encoded representations.If the system caches encoder outputs or key/value blocks, the KV cache behaves just like any other cached token or cached KV entry. Router and cache mechanisms that locate conversation prefixes will find image-derived prefixes just the same. This is often where the most expensive work is saved.
A quick honest note: the Qwen 0.5B model used in this project reads text only, so the multimodal discussion above is conceptual for that particular model. Multimodal serving is nevertheless part of the project’s public work on real accelerators and integrates with the same infrastructure patterns.
Both stories—agents that re-read history at every step, and images that expand a single request into many pages of reading—converge on the same solution: the cache. When repeated prefixes are saved, the system avoids redoing the most expensive work.Now, let’s watch that conclusion unfold on our cluster: one agent-shaped workload, eight turns, live.
