KV cache and batching: why a fast AI service can keep you waiting

A server can produce more text per second while making some users wait longer. Overall throughput, the delay before the first piece of the response and the time needed to receive the complete answer are different measures. The KV cache and request batching affect these measures, but do not improve them all in the same way across every configuration.

These concepts are particularly relevant to serving autoregressive language models, which build their output progressively. They help you read a performance test without confusing computing power with the quality of the waiting experience.

Four clocks for a single response

For the user, the delay starts when they send their request. On the server, several stages may follow: waiting in a queue, processing the input text, generation and transmission. A measurement limited to model computation leaves out part of this journey.

Measure	Question it answers	What it does not describe on its own
Time to first token	When does the response begin?	When will it be complete?
Time between tokens	Does the text arrive steadily?	How long was the wait before the first token?
Total duration	When is the complete response available?	How much of the duration comes from its length?
Service throughput	How much output does the system produce over a period?	The delay experienced by each person

A token is a unit of text processing, not necessarily a word. The guide to tokens and context windows (in French) explains the difference. Comparing throughput across models or languages therefore requires looking at what was actually produced.

The KV cache retains work already done

In the attention mechanism of an autoregressive Transformer, keys and values, often abbreviated K and V, represent information used to process previous positions. The KV cache retains these elements to avoid recalculating them for every new token. It consumes memory and holds state during generation. The Transformers documentation on caches describes several strategies: a dynamic cache, fixed allocation, offloading to processor memory or reduced precision.

This cache is not a persistent corporate knowledge base. It guarantees neither an accurate response nor up-to-date documents. Nor should it be confused with a cache of complete answers that returns a previously computed result.

There are two capacity questions: retaining the information needed for each sequence and serving several sequences simultaneously. The original work on PagedAttention specifically addresses this memory management problem for serving large models. It shows why cache organisation can limit the number of requests handled. Its experimental results apply to the study’s configurations, not to a guaranteed speedup for every deployment.

Grouping requests changes server utilisation

A batch allows several requests to be processed together. In a fixed batch, requests do not necessarily have the same length: some finish while others continue. Continuous batching rebuilds the group during generation, admitting new requests as others finish. This is the principle described in the dedicated Hugging Face documentation.

This organisation aims to make better use of available resources. But a rule that waits for more requests before starting work can lengthen an individual’s wait. A scheduling policy should therefore be examined under load, with real differences in length, not just an identical set of requests arriving at the same instant.

You can picture the problem without assuming how a particular program behaves: a short request arrives while a long one is already running. The service must decide when to prepare the new input and how to share resources with outputs in progress. The outcome depends on its rules and available capacity.

Higher throughput does not mean a smoother conversation

Take two entirely fictional services over one minute, with the same requests and outputs of equal length. A completes 60 responses and B completes 80. B has higher throughput over that window. Suppose, however, that the time to first token for the request being examined rises from 0.4 seconds with A to 1.2 seconds with B. For that person, the response begins later.

This example is logically possible; it describes no real measurement. It illustrates that an aggregate figure does not contain the distribution of waiting times. To compare the services, accompany averages with quantiles, such as the delay below which 95% of observations fall, specifying the size and nature of the sample.

Also count rejected or interrupted requests. A service that ignores difficult requests can display attractive latency for those it accepts. The protocol should report requests sent, completed, failed and still waiting at the end of the window.

Memory savings can carry a cost elsewhere

Cache strategies involve implementation trade-offs. Transformers notes, for example, that a quantised cache can reduce memory use but worsen latency in some cases; offloading the cache to the processor adds transfers. Fixed allocation can facilitate certain optimisations while reserving unused space. These documented properties do not rank the strategies for your particular workload.

The guide to memory and computation in AI chips (in French) provides the hardware context. Here, the practical decision is to measure which constraint dominates for the input lengths, output lengths and numbers of simultaneous requests expected for the service.

Changing cache precision also warrants checking the outputs. A performance optimisation is not implicit permission to change the expected quality level. Retaining the engine version and its settings makes the result interpretable when an update changes the trade-offs.

The minimum protocol for a useful demonstration

Prepare several request profiles: short input and short output, long input and short output, long generation, then a representative mixture. Fix the model, output limits and success criteria. Increase load gradually and record the four measures in the table at each level.

Distinguish a cold start from a service already loaded into memory. Document any prefixes shared between requests and how they are cached. Keep the same measurement point, ideally at the client when user experience is the objective. Internal timings remain useful for diagnosis but do not replace the complete journey.

The relevant deliverable is therefore not simply “this many tokens per second”. It is a service curve: at what load responses remain correct, within what delay, with what failure rate and using which resources. This lets you choose a configuration for a use case without turning a technical optimisation into a universal promise.

Sources and scope

Consulted on 22 October 2026: Transformers, cache strategies and continuous batching, evolving documentation; Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, 2023. The tables and protocol are an educational synthesis; the latency example is fictional.