Serving the agentic workload: Simon Mo on what AI agents change in vLLM

At Runtime 2026, Modal’s conference for engineers running AI in production (1 October 2026, The Midway, San Francisco), Simon Mo explained how AI agents change the job of an inference server and what vLLM does about it. Simon is co-founder and CEO of Inferact, the company founded by vLLM’s creators and maintainers.

Modal published the 18-minute talk, “Serving the Agentic Workload”, on 2 November 2026. This is vLLM Bangkok’s summary: every number has a source, and our own additions are marked.

This is our 20-slide English summary deck; a Thai version is on the Thai post.

Open PDF20 pages · 0.2 MB

TL;DR

1. The workload changed. A coding-agent session reads a very long context (median 142k tokens in), writes little (444 tokens out) and loops for dozens of turns.
2. The KV cache is the core. vLLM answers with a hybrid KV cache, prefill/decode disaggregation and a distributed KV cache pool. The pool alone, built on Mooncake, gave 3.8x higher throughput on Codex agentic traces (1P1D, 12x GB200).
3. Hardware is now plural. NVIDIA Blackwell and AMD MI355X are at production parity. TPU runs Gemini and Anthropic’s models, and Inferact’s experimental TPU megakernel reached 709 tok/s for one user on Kimi K3 (16x TPU v7, speculative decoding).

Our one-line version: in the agent era, speed comes less from a faster GPU than from not recomputing.

The workload changed

In Simon’s words, “vLLM is a large language model inference and serving engine.” It began as serving research at UC Berkeley’s Sky Computing Lab (see the vLLM docs), and its first prototype, he says, predates ChatGPT. The talk’s slide counted 3,488 contributors to the repository (GitHub, September 2026).

Three eras of workload:

• 2020-2022, single-turn completion: “one prompt auto complete”. vLLM’s first API was completions, not chat.
• 2022-2024, multi-turn chat: every turn carries the history, so vLLM built automatic prefix caching, chunked prefill and prefill/decode disaggregation.
• 2024-2026, agentic and real time: tool calls, sub-agents and real-time interaction, “just a blow up of all the complexities”. Serving it is “a big focus of vLLM this year”.

Anatomy of a coding-agent session

The key slide draws one session as stacked turns. Turn 1 is a fresh prefill; each later turn appends thinking, tool calls and tool results, takes the old prefix from cache and prefills only the new part. By turn 43 almost the whole prompt is cached: “cached prompt skips recompute”. One tool call can fan out into many parallel sub-agent sessions.

Beside it: 43 turns per session, P50 input 142k tokens, P50 output 444 tokens, above 95% prefix reuse and 44% bursty sub-agents. Per the vLLM x AgentX post (8 September 2026), these are statistics of SemiAnalysis AgentX, “a public benchmark built from real-world agentic coding traces”; Simon framed them with Inferact’s experience serving customers but did not call them Inferact’s traffic. The prefix figure is theoretical reuse (“a theoretical cache hit rate of more than 96%”), not a measured hit rate, and 44% is the share of sessions with at least one sub-agent.

Read 142,000 tokens to write 444

That is about 320 tokens read for each token written (our arithmetic on two medians; the traces in the vLLM x Mooncake post average 131:1). Real speed depends on how much context the engine can reuse instead of recompute.

Primer: prefill, decode and the KV cache (our addition)

Not from the talk; just the terms.

• Prefill processes the whole input (prompt, history, tool output) at once. It is compute-heavy and sets the time to first token (TTFT).
• Decode writes one token at a time. Usually memory-bandwidth bound, especially at small batch sizes, it sets tokens per second per user.
• The KV cache stores the attention keys and values already computed, in prefill and decode. A request that shares the prefix reads them back instead of recomputing: a prefix cache hit. It eats GPU memory, more so with long contexts and many users.

Where it gets hard

Now “multiply that by hundreds, thousands” of users, where a session “could be 10 turns, it could be 100 turns, it could have a 100K token tool call or tool call result”. The slide names three problems:

1. Prefix cache pressure: “Every call replays the full context, so KV cache reuse carries the load.”
2. Execution efficiency: “Each step processes more tokens while staying inside a tight latency budget.”
3. Global routing: “Routing sessions and bursty sub-agents means trading cache affinity against load balance.”

What vLLM does about it

Hybrid KV cache: one page size for all attention types

PagedAttention was designed for dense models. Hybrid and sparse models mix layers whose caches behave differently: full attention grows with tokens, sliding window is fixed per request, and linear attention is a constant size, “although a little bit large”.

vLLM picks one page size, the least common multiple of the types’ sizes (LCM(X, Y, Z) on the slide), so every layer shares one pool and any free page fits any layer. The LCM allocator is described in the Jenga paper; the hybrid KV cache manager design doc describes the single pool. In production, Simon said, “we’re seeing typically 2x or 2.5x more KV cache space that’s getting freed up, and therefore being able to deliver end-to-end throughput and cost savings.” That figure is spoken, not published.

Prefill/decode disaggregation

“Nobody is serving production agentic workload on an aggregated inference setup.”

In an aggregated setup one server does both phases, and “it delivers inconsistent performance”: heavy prefill makes a streaming answer pause mid-thought, and heavy decode blocks new prefills, so “your TTFT absolutely suffers”. Separate prefill and decode instances joined by a distributed KV cache pool are, Simon said, “the only way to go for large scale inference and production inference” in 2026. DistServe makes the research case. Our addition: each side can also be sized on its own.

A distributed KV cache pool with Mooncake

vLLM has native integrations with a few frameworks and providers; the talk showed the Mooncake connector. In the control plane, vLLM schedulers look up a Mooncake Master that tracks where KV blocks live. In the execution plane, each vLLM instance runs a vLLM Worker and a Mooncake Client beside its GPU HBM and moves blocks over GPUDirect to the data plane: a distributed KV cache pool on an RDMA fabric, backed by host DRAM and NVMe SSD. A prefix computed on one machine can be reused on another.

The slide claims 3.8x higher throughput and 46x lower TTFT. Per the Mooncake post, that is Kimi-2.5 (NVFP4), 1P1D on 12 GB200 GPUs and 610 real Codex and SWE-bench Pro agentic traces, against a baseline caching only the system prompt: cache hits rose from 1.7% to 92.2%, P50 TTFT fell 46x and end-to-end latency 8.6x, using CPU DRAM (the NVMe tier came later).

Reading the Pareto chart

Results are compared on SemiAnalysis InferenceX, common ground shared with NVIDIA, AMD and Google. The slide’s chart, “Kimi K3 2.8T Agentic Token Throughput per Chip vs. P90 Interactivity” (updated 21 September 2026), plots GB300 NVL72, GB200 NVL72, B200, B300 and MI355X (vLLM and Dynamo series). How to read it:

• Y-axis: throughput per chip. Higher is more economical.
• X-axis: interactivity, tokens per second per user (P90). Further right is faster per user.
• Pareto frontier: “You cannot get best of both world.” Top left is the most economical setup; bottom right is the fastest, where “you can just throw the GPU just for that single user”.
• Each point is one configuration plus kernel-level optimisation.

vLLM’s job is to “push the curve as much as possible to the top right” and let you pick any point.

What’s next

Agentic-first: agent/session hints and programmable KV caches, so “you can set TTL, you can actually set priorities, you can figure out where KV cache should live”. Simon’s example: a session you know you will not touch for an hour. Pin it, let LRU decide, or evict it? The Q4 roadmap lists cache-tier TTLs and request-level control of prefetching and eviction.

Agentic-efficient: ultra-low latency (speculative decoding, engine efficiency) and instant scaling and recovery, so that “you can pick any points on the Pareto curve and it’ll automatically run in production for you.”

Agent-friendly: interfaces and infrastructure for agents that develop vLLM. The team already optimises vLLM with models served by vLLM, in a full loop with agentic evaluation.

Hardware heterogeneity

Simon called this the year of hardware heterogeneity. First, “you finally have parity of chips”: NVIDIA Blackwell and AMD MI355X have similar specs and availability and deliver production performance, and “more and more” large vLLM users deploy on AMD.

• NVIDIA Blackwell: “general purpose, well designed, battle tested.”
• AMD MI355X: big HBM, 288 GB of HBM3E at 8 TB/s, the same as B300 (per the AMD and Embedded LLM post).
• Google TPU v7 Ironwood: a giant systolic array and a large, programmable VMEM. TPUs underpin Anthropic’s models and Gemini.

To see what open source can do on TPU, Inferact “recently built some experiments”.

The TPU megakernel

From the slide “First TPU Megakernel: 709 tok/s/user”, in tokens per second per user:

• Kimi K3: 709 on 16x TPU v7 vs 452 for the vLLM baseline on 16x GB200, about 1.6x.
• Qwen 3.8 27B: 1,515 on 4x TPU v7 vs 695 on 4x GB200, about 2.2x.

The multipliers match Inferact’s own charts (1.57x and 2.18x), and these are best-case, per-user figures measured by Inferact. Inferact’s write-up gets “over 700 tokens/s with speculative decoding” (DSpark, one user) against vLLM’s published Kimi K3 recipe on 16 GB200, and its acceptance-length chart shows the GB200 baseline also ran DSpark speculative decoding (452 tok/s at acceptance length 6, 229 at 3). Without it, Inferact reports roughly 1.4 to 2x at batch sizes 1 to 8. The Qwen numbers are not in the write-up’s text, but its headline chart shows them too (1,515 vs 695 on 4 chips each). Inferact calls the work “a first step” and claims, to its knowledge, “the first open-source inference megakernel written in Pallas”, which is narrower than the slide’s “First TPU Megakernel”; Google’s TPU engines announcement reports the same single-user result.

Simon’s point: “this is not just like benchmarking”. The aim is “to understand the core differences between each hardware.”

What VMEM prefetching buys

One kernel manages a whole decode step: all 92 MoE layers of Kimi K3 run in a single Pallas call. Separate kernels load from HBM, compute and store in turn, leaving the bus idle in the gaps. TPU’s VMEM, “more or less equivalent to the shared memory on GPU”, is “a lot bigger and a lot more programmable”. So the kernel prefetches the next layer’s attention and MoE weights while the current layer computes, pipelining Simon called “not doable today on NVIDIA’s PDL”.

Premature abstraction is bad

One lesson the team learned: “this idea of premature abstraction, right, is bad.” In 2024 and 2025 they abstracted kernel and model interfaces, “aiming to unify and simplify” across “thousands and thousands of model architecture and hundreds and more of hardware”. In 2026, for extreme performance, they split instead: “every single model is getting their own file, their own kernel, and their own essentially inference engine”, on vLLM as the shared core runtime. Agents write more kernels, state-of-the-art kernels come in from the hardware teams, and model structure stays flat.

vLLM Q4 roadmap

The slide “vLLM Q4 Roadmap: Agentic Production Serving” has four fronts, detailed in the roadmap issue:

• Production: agentic workloads, large-scale serving.
• Performance: model performance and speculative decoding, which “continues to be the method to push latency further” at low concurrency.
• Refinement: KV cache refactoring, plus CI and release, to keep vLLM reliable and secure.
• Ecosystem: hardware support, RL and omni-modal models.

Each has a working group open to everyone, meeting on the public vLLM calendar. “We never found a harder problem to work on than keeping the project at this scale sustainable and useful for everybody,” Simon said. To start: docs.vllm.ai, vllm.ai/blog and the GitHub repository.

Takeaways for teams running their own models (ours)

Our suggestions, not the talk’s.

1. Measure first: input/output ratio, turns per session and prefix cache hit rate of your own agent traffic.
2. Keep the prefix stable: system prompt, tool definitions and static documents first, with no timestamps or other per-request values in them.
3. Choose your Pareto point: batch jobs want throughput per chip; interactive jobs want tokens per second per user.
4. Plan beyond one node: prefill/decode disaggregation and a KV cache connector such as Mooncake or NIXL, which pay off mainly at scale.

Sources and references

1. Serving the agentic workload | Simon Mo CEO at Inferact, Runtime 2026 — Modal on YouTube, 2 November 2026. The talk: every quote and slide figure above.
2. Runtime Recap — Modal, event held 1 October 2026. Event name, date and venue.
3. [Roadmap] vLLM Roadmap Q4 2026 — vllm-project/vllm on GitHub (issue 61024), 10 October 2026. Cache-tier TTLs and request-level eviction/prefetch control, the per-SIG Q4 plans behind the slide’s four fronts, and SIG meetings on the public vLLM calendar.
4. Hybrid KV Cache Manager — vLLM documentation. One memory pool with a single page size for all layer types.
5. Jenga: Effective Memory Management for Serving LLM with Heterogeneity — arXiv (Zhang, Du et al.), 24 March 2025. The LCM-based page allocator.
6. Serving Agentic Workloads at Scale with vLLM x Mooncake — vLLM Blog, 6 May 2026. The 3.8x, 46x and 8.6x results and their setup; the Mooncake architecture; the 131:1 trace ratio.
7. kvcache-ai/Mooncake — GitHub. Mooncake Store, its RDMA transfer and its DRAM and SSD/NVMe tiers.
8. vLLM x AgentX: Optimizing for Real-World Agentic Serving — vLLM Blog (vLLM Team and Inferact), 8 September 2026. The session statistics and their origin in SemiAnalysis AgentX traces; NVMe tier added later.
9. AI Accelerator System Benchmark | AcceleratorX by SemiAnalysis — SemiAnalysis (site header: InferenceX). The benchmark behind the Pareto chart; the 21 September chart itself is from the talk slide.
10. Scaling Kimi K3 on AMD Instinct MI355X: Matching NVIDIA HGX B300 Throughput per GPU on AgentX — vLLM Blog (AMD and Embedded LLM), 16 October 2026. MI355X and B300 memory specs.
11. 700 TPS on Kimi K3: A Case for TPU Megakernels — Inferact News, September 2026. Megakernel setup and baseline (both with DSpark speculative decoding), the roughly 1.4 to 2x without it, the Qwen numbers in its headline chart and the Pallas claim; 92 MoE layers in one Pallas call; cross-layer weight prefetch.
12. Announcing New vLLM and SGLang Inference Engines on TPU — Google Developers Blog, 20 October 2026. The single-user result on Ironwood; Inferact as founded by vLLM’s creators.
13. Efficient Memory Management for Large Language Model Serving with PagedAttention — arXiv (Kwon et al., SOSP 2023), 12 September 2023. The original PagedAttention design for dense models.
14. Welcome to vLLM — vLLM documentation. vLLM’s origin in the Sky Computing Lab at UC Berkeley; the Build link.
15. Blog | vLLM — vLLM. The Learn link from the closing slide.
16. vllm-project/vllm — GitHub. vLLM as an inference and serving engine for LLMs; the Contribute link.
17. Inferact | Frontier Inference You Control — Inferact. Inferact as the company founded by vLLM’s creators and maintainers.
18. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — arXiv (Zhong et al.), 18 January 2024. Why splitting prefill and decode removes interference and lets each phase get its own resources.

Further watching

• Serving vLLM on Agentic Production Workloads | Inferact | Ray Summit 2026 — Anyscale on YouTube, 17 September 2026 (37 minutes).
• vLLM and the State of AI Inference | Simon Mo (Inferact) | Ray Summit 2026 — Anyscale on YouTube, 17 September 2026 (19 minutes).
• Our Ray Summit 2026 Recap page — the decks from our 24 October 2026 recap in Bangkok.

A community summary by vLLM Bangkok. Quotes are from the talk unless another source is named; corrections are welcome via the contact on this site.