Instrument 10 / inference cache

Context length has a memory bill.

Calculate KV-cache memory per token, per sequence, and across concurrent requests—then compare grouped-query attention with a full multi-head cache.

Size one cache

Make context and concurrency explicit.

Choose a sourced model preset or enter the exact configuration. Nothing is submitted or stored; every calculation runs in this browser.

Model architecture

Model presetPresets copy architecture values; every field remains editable. Transformer layersnum_hidden_layers in many model configs. layers Attention headsTotal query heads, used for the MHA comparison. heads Key-value headsUse num_key_value_heads, not total query heads. heads Head dimensionUse the configured value or hidden size divided by attention heads. dim

Serving workload

Cached contextFull cached tokens held by each live sequence. tokens Concurrent sequencesSimultaneous full caches, not prompt-processing microbatch size. seq KV-cache precisionConfirm the exact runtime supports the selected cache format. VRAM reserved for KVMemory left after weights and other runtime allocations. GiB

Sourced architecture presets

Same 32K context. Different cache shapes.

These scenarios use 16-bit KV cache, one full-context sequence, and architecture fields from the publishers' model configurations. Open one, then change context and concurrency.

Qwen2.5 7B · 32K 1.75 GiB 56 KiB per token · 7× GQA reduction → Qwen2.5 32B · 32K 8 GiB 256 KiB per token · 5× GQA reduction → Qwen2.5 72B · 32K 10 GiB 320 KiB per token · 8× GQA reduction →

Cache boundaries

Full context is a ceiling.
Real requests may hold fewer cached tokens at a given moment.

Concurrency multiplies.
Each independent full sequence needs its own cache unless prefixes are shared.

Allocation still matters.
Paged, sliding-window, chunked, and offloaded caches change residency and waste.

Transparent method

What this KV-cache calculator computes.

Bytes per token
2 × layers × KV heads × head dimension × bytes per element

Bytes per sequence
bytes per token × cached context tokens

Total cache
bytes per sequence × concurrent sequences

GQA comparison replaces KV heads with total attention heads to show the cache a full multi-head-attention architecture would use at the same layer count, head dimension, context, concurrency, and precision.

The arithmetic describes a dense full cache. It does not add model weights, activations, CUDA context, allocator fragmentation, metadata, block rounding, or runtime workspace. Use the LLM GPU Memory Calculator for the combined capacity plan.

Source the inputs

Read the model config before trusting the result.

Architecture names vary. Confirm hidden layers, attention heads, KV heads, head dimension, cache format, and any hybrid or sliding-window behavior in the exact model and runtime you plan to serve.

Review the Transformers KV-cache guide → Inspect the Qwen2.5 7B config → Inspect the Qwen2.5 32B config → Inspect the Qwen2.5 72B config → Inspect the Mistral 7B v0.3 config →

How do you calculate LLM KV cache memory?

Multiply two for keys and values by layers, KV heads, head dimension, cached tokens, concurrent sequences, and bytes per cache element. Divide bytes by 1,073,741,824 for GiB.

Why do grouped-query attention models use less KV cache?

Grouped-query attention shares key and value heads across multiple query heads. Fewer KV heads means fewer stored elements per token than full multi-head attention.

How does context length affect KV cache memory?

A dense full cache grows linearly with cached tokens and concurrent sequences. Double either input and the calculated memory doubles.

Does KV cache memory include model weights?

No. This result isolates the cache. Add weights and deployment-specific runtime allocations before deciding whether the full workload fits.