LLM infrastructure / 70B worksheet

70B does not have one VRAM number.

Q: How much VRAM does a 70B model need at 4-bit precision?

Seventy billion parameters at exactly four bits require about 32.6 GiB for raw weights. A serving deployment needs more for format overhead, runtime allocations, activations, and KV cache. In the illustrative 32K-context example on this page, the planning target is 51.1 GiB.

Q: Can a 70B model run on one 48 GB GPU?

A simple 4-bit weight floor can fit within 48 GB, but that does not prove the serving workload fits. In this page's illustrative architecture with a 32K 16-bit KV cache, 20 percent runtime headroom, and 90 percent usable VRAM, the estimate needs two 48 GiB GPUs.

Q: How much VRAM do 70B FP16 weights require?

Seventy billion parameters multiplied by two bytes per parameter is about 130.4 GiB for weights alone. KV cache and runtime overhead increase the serving requirement.

Weight precision sets the floor. Architecture, cached context, concurrency, runtime headroom, and actually usable memory decide whether the serving workload fits.

Open the editable 70B model →

The weight floor

Start with bytes. Do not stop there.

INT4 weights
70B × 4 bits ÷ 8 = 35 GB = 32.6 GiB

INT8 weights
70B × 8 bits ÷ 8 = 70 GB = 65.2 GiB

FP16 weights
70B × 16 bits ÷ 8 = 140 GB = 130.4 GiB

Serving target
(weights + explicit KV cache) × (1 + runtime headroom)

Illustrative 32K serving profile

Hold the architecture constant. Change the weights.

Each editable scenario uses 80 layers, eight KV heads, 128 dimensions per head, 32,768 cached tokens, one concurrent sequence, 16-bit KV cache, 20% runtime headroom, and 90% usable VRAM. Those are example inputs—not properties of every 70B model.

INT4 · 2 × 48 GiB 51.1 GiB 32.6 weights + 10.0 KV + headroom → INT8 · 2 × 80 GiB 90.2 GiB 65.2 weights + 10.0 KV + headroom → FP16 · 3 × 80 GiB 168.5 GiB 130.4 weights + 10.0 KV + headroom →

Deployment boundaries

Fit is not throughput.
Enough capacity does not prove useful tokens per second.

One sequence is not a service.
KV cache grows linearly with full concurrent caches.

Topology matters.
The GPU count must match a supported parallel configuration and interconnect.

Read the estimate

Change the architecture before changing the hardware.

Copy layer count, KV heads, and head dimension from the exact model configuration. Set the cached context and concurrent sequences from the serving target, then enter the memory you can actually allocate on each device.

Open the LLM GPU memory calculator → Inspect the 10 GiB KV-cache calculation → Read ResearchAudio's GPU architecture guide → Check current Transformers cache strategies →

How much VRAM does a 70B model need at 4-bit precision?

Exactly four bits per parameter is about 32.6 GiB for raw 70B weights. The illustrative 32K-context serving profile on this page reaches 51.1 GiB after explicit KV cache and 20% runtime headroom.

Can a 70B model run on one 48 GB GPU?

A 4-bit weight floor can fit, but the serving workload may not. With this page's explicit 32K KV cache, headroom, and 90% usable-memory assumption, the estimate needs two 48 GiB GPUs.

How much VRAM do 70B FP16 weights require?

Seventy billion parameters at two bytes each is about 130.4 GiB for weights alone. KV cache and runtime allocations increase the serving requirement.

Why is a model file smaller than the serving-memory requirement?

Serving also allocates KV cache, activations, kernels, workspace, and framework state. Quantized files may include scales, metadata, and higher-precision modules.