KV cache: calculate the memory behind each conversation.

What attention keeps between steps

In a conventional causal transformer, each attention layer projects a token representation into a query, a key, and a value. The query determines which earlier keys matter; the corresponding values contribute to the layer's output.

Caching the earlier keys and values avoids reconstructing them at each generation step. This is inference state, not an update to the model's weights or a permanent memory of the conversation.

For unchanged inputs and execution configuration, later tokens do not change the causal state of earlier positions. Layers are evaluated in dependency order; prompt positions can be processed together within a layer.

Derive the memory budget

For ordinary multi-head or grouped-query attention: KV bytes = 2 × L × Hkv × D × S × B × e. L is layers, Hkv KV heads, D head dimension, S retained positions, B independent sequences, and e bytes per cache element.

With L = 32, Hkv = 8, D = 128, and e = 2, each position costs 131,072 bytes. An 8,192-position sequence therefore uses 1GiB. Four independent sequences require 4GiB before allocation overhead.

Grouped-query attention reduces Hkv by sharing key/value heads among query heads. The GQA paper explains that architectural choice. Query-head count is not interchangeable with KV-head count in this calculation.

The calculator below uses its selected model configuration. Treat the memory budget as space remaining after weights, runtime buffers, services, and safety headroom—not all installed RAM.

Know where the formula stops applying

The formula describes an uncompressed cache with equal-length sequences. Real requests have different lengths, and an allocator may reserve blocks beyond the live token count. Budget for expected output growth as well as the prompt.

Sliding-window attention retains only part of the history in affected layers. Latent attention and hybrid architectures use other state layouts.

Cache quantization reduces element storage while adding its own metadata and accuracy considerations.

The animation illustrates ordinary append-only growth. It does not model eviction, sliding windows, speculative rollback, or a cache that has been compressed or moved between devices.

Separate a sequence cache from a prefix cache

A sequence cache serves an active generation. A prefix cache can retain eligible blocks after that request ends and reuse them for a later request with the same beginning.

Request completion does not necessarily mean all its state has been discarded.

A text edit invalidates state from the first changed token onward in the simple causal model. Changes to model weights, adapters, tokenization, or other computation settings can invalidate reuse even when the visible text is unchanged.

Stable instructions first and variable material later improve opportunities for reuse. The cache key must also identify the relevant model configuration and security scope. A matching string alone is not a sufficient production cache key.

Verify isolation and observe pressure

Measure allocated cache bytes, active positions, hit rate, eviction, and rejected requests. PagedAttention addresses fragmentation and sharing, but it does not create unlimited capacity.

Keep cache reuse inside an authorized trust group. vLLM documents per-request cache salts for this purpose. Set that scope from trusted application identity, and include cache state in retention and deletion procedures.

Load-test long prompts and long outputs together. A service that handles many short requests may exhaust its cache when a few users open large documents.