Instrument 09 / GPU fit

Find out if the model fits before renting the GPUs.

Turn parameter count, weight precision, architecture-aware KV cache, runtime headroom, and usable VRAM into a transparent minimum-GPU estimate.

Size one deployment

Convert model weights into a VRAM plan.

This is an early inference estimate, not a benchmark. Nothing is submitted or stored; every calculation runs in this browser.

Weights

Model parametersUse the total parameter count, not active MoE parameters. B Weight precisionSimple bits-per-parameter estimate before format overhead. Exact checkpoint sizeOptional. When known, this artifact size replaces the bits-per-parameter weight estimate. GiB

Optional KV cache

Transformer layersUse zero to omit explicit KV-cache memory. layers Key-value headsUse KV heads, not total attention heads, for grouped-query attention. heads Head dimensionPer-head key or value dimension from the model configuration. dim Cached contextMaximum cached tokens per live sequence. tokens Concurrent sequencesSimultaneous sequences holding a full KV cache. seq KV-cache precisionBytes stored for every cached key and value element.

Runtime and hardware

Runtime headroomPlanning allowance for activations, kernels, workspace, and format overhead. % VRAM per GPUUse the capacity of the exact accelerator you can deploy. GiB Usable VRAMLeave room for the runtime, fragmentation, and non-model allocations. % GPUs availableThe hardware count in the deployment you are checking. GPU

Sizing boundaries

Weights are the floor.
A file fitting in VRAM does not prove the serving workload fits.

Context consumes memory.
KV cache grows with architecture, sequence length, batch size, and concurrency.

Profile the runtime.
Tensor parallelism, kernels, and quantization formats add deployment-specific overhead.

Transparent method

What this LLM GPU memory calculator computes.

Weight memory uses the exact checkpoint size when you provide one. Otherwise, it multiplies total parameters by bits per parameter, divides by eight, and converts bytes to GiB. It deliberately uses total parameters for sparse models because inactive experts still have to be stored unless the deployment explicitly offloads them.

KV-cache memory is 2 × layers × KV heads × head dimension × context tokens × concurrent sequences × bytes per KV element. The factor of two stores keys and values. Sliding-window, chunked, quantized, offloaded, or shared-prefix caches need runtime-specific treatment.

Planning target adds the runtime headroom you choose after weights and any explicit KV cache. Hugging Face's model memory estimator describes weight loading as a floor and notes that inference can require additional memory. Use measured runtime data instead of the default once you have it.

Minimum GPUs divides the planning target by usable VRAM per GPU and rounds up. This capacity check does not promise throughput, latency, interconnect efficiency, or compatibility with a particular tensor-parallel configuration.

Quantized formats can include scales, metadata, and higher-precision modules. The calculator therefore treats bit width as a planning input, not a guarantee that every 4-bit or 8-bit artifact occupies exactly the same amount of memory.

GPU sizing guide

Four questions before you call the model deployable.

Memory fit is necessary, but it is only the first gate. Validate the exact model artifact, context window, batch profile, serving runtime, and hardware topology.

Compare 70B INT4, INT8, and FP16 scenarios → Compare Qwen3 dense and MoE memory targets → Calculate KV-cache memory and GQA savings → Read ResearchAudio's GPU architecture guide → Check the Hugging Face model memory estimator → Review current Transformers KV-cache strategies → Review current Transformers quantization support →

How much GPU memory does an LLM need?

Start with parameter count multiplied by bits per parameter. Then add capacity for runtime overhead, activations, and the KV cache. A weight-only estimate is a lower bound, not a production guarantee.

How many GPUs does a 70B model need?

At 16-bit precision, 70 billion parameters require about 130 GiB for weights before inference overhead. The final GPU count depends on usable VRAM per device, quantization, context length, concurrency, and the serving runtime.

How does quantization change LLM VRAM?

Lower-bit formats reduce weight memory, but real artifacts can also store scales, metadata, and some modules at higher precision. Treat the bit-width result as an estimate and check the actual model artifact.

Does this calculator include KV cache memory?

Yes, when you provide layers, KV heads, head dimension, context length, concurrent sequences, and KV precision. Leave the architecture fields at zero for a weight-plus-headroom floor.