Instrument 15 / local inference

Can your GPU run the local LLM?

Choose a model size, weight format, and real GPU memory profile. Get the transparent weight-floor fit first, then add architecture and context before calling the deployment viable.

Check the hardware →

First gate / weight floor

Check the checkpoint before the download.

Every calculation stays in this browser. This first pass excludes KV cache; the result links into the architecture-aware calculator with your selections preserved.

Model sizeTotal parameters stored, not active MoE parameters. Weight precisionPlanning bit width before artifact-specific metadata. GPU memory profileCapacity per device; vendor and runtime support still matter. GPU countAggregate capacity only; topology can restrict real sharding. GPU Runtime headroomReserve for activations, kernels, workspace, and format overhead. Usable GPU memoryLeave space for the runtime and fragmentation.

Interpret the result

Fit starts with weights.
Quantized artifacts add metadata and sometimes preserve layers at higher precision.

Context is separate.
KV cache depends on layers, KV heads, head dimension, sequence length, and concurrency.

Capacity is not speed.
Bandwidth, kernels, offload, and interconnect decide delivered tokens per second.

Static planning matrix

Model size × precision × first viable profile.

Each cell adds 20% runtime headroom, assumes 90% usable GPU memory, and excludes KV cache. “First profile” means the smallest capacity option on this page that clears the arithmetic floor.

Swipe the matrix horizontally →

Model	INT4	INT8	FP16 / BF16
7B	3.91 GiB8 GB profile	7.82 GiB12 GB profile	15.65 GiB24 GB profile
13B	7.26 GiB12 GB profile	14.53 GiB24 GB profile	29.06 GiB48 GB profile
32B	17.88 GiB24 GB profile	35.76 GiB48 GB profile	71.53 GiB80 GB profile
70B	39.12 GiB48 GB profile	78.23 GiB141 GB profile	156.46 GiBMulti-GPU only

A GPU profile clearing this table can still fail after the exact checkpoint size, KV cache, runtime workspace, unsupported quantization kernels, or tensor-parallel constraints are included.

Capacity references

The presets use published GPU memory, not guessed capacity.

The branded examples are capacity anchors. Board-partner variants and runtime support can differ, so confirm the exact card and software stack before buying or renting hardware.

Profile	Reference hardware	Usable at 90%
8 GB	Generic consumer GPU profile	7.2 GiB
12 GB	NVIDIA GeForce RTX 3060	10.8 GiB
16 GB	Generic consumer or workstation profile	14.4 GiB
24 GB	NVIDIA GeForce RTX 4090	21.6 GiB
32 GB	NVIDIA GeForce RTX 5090	28.8 GiB
48 GB	NVIDIA RTX 6000 Ada	43.2 GiB
80 GB	NVIDIA A100 80GB / H100 SXM	72 GiB
141 GB	NVIDIA H200	126.9 GiB

Transparent method

What “fits” means here—and what it does not.

Weight memory is total parameters × bits per parameter ÷ eight, converted to GiB. The checker adds the runtime headroom you select, then compares that target with the selected fraction of GPU memory.

Minimum GPUs is an aggregate capacity division rounded up. It does not promise that the runtime supports the required tensor, pipeline, or expert parallel layout.

KV cache is deliberately excluded because parameter count alone cannot reveal it. Open the full calculator and provide the exact layer count, KV heads, head dimension, context, concurrency, and cache precision.

CPU or unified-memory offload can make a larger artifact runnable, but changes latency and throughput. This checker answers the narrower fully GPU-resident capacity question first.

Open the architecture-aware GPU memory calculator → Know the GPU instead? Find the largest model-size floor → See 7B and 13B 32K-cache examples → See a 70B architecture-aware worksheet → Review the Hugging Face model memory estimator → Review Transformers KV-cache strategies →

Local LLM compatibility FAQ

Capacity questions before the runtime benchmark.

Use these answers as arithmetic boundaries. Validate the exact artifact and runtime before treating a profile as a hardware recommendation.

How do I know if my GPU can run a local LLM?

Compare the model's weight memory, runtime headroom, KV cache, and usable GPU memory. This checker answers the weight-floor gate; the full calculator adds the architecture and context inputs.

Can a 24 GB GPU run a 70B LLM?

Not as a fully GPU-resident 70B INT4 deployment under the default assumptions here. The weight-plus-20%-headroom target is about 39.12 GiB before KV cache. CPU offload or a different artifact changes the boundary and performance.

How much VRAM do 7B and 13B models need?

With 20% headroom and no KV cache, 7B planning floors are 3.91, 7.82, and 15.65 GiB at INT4, INT8, and FP16. The 13B floors are 7.26, 14.53, and 29.06 GiB.

Why does a memory fit not guarantee fast local inference?

Capacity and speed are different constraints. Tokens per second also depend on memory bandwidth, compute support, runtime kernels, quantization format, offload, context, batching, and multi-GPU communication.