How Much VRAM Does Your LLM Actually Need? A Field Guide to Sizing GPUs¶

"Will this model fit on my GPU?" has one honest answer: do the arithmetic. It's three numbers added together, and you can pull every input from a model's config.json in about two minutes. This post turns a sizing spreadsheet into a method you can run by hand.

Most "VRAM calculators" hide the math behind a slider. The math is the useful part, because once you see it you can size any model, at any context length, on any card, without trusting a black box.

The whole thing in one line¶

For inference, GPU memory is just:

Total VRAM = (Model Weights + KV Cache) / 0.9

The / 0.9 is because vLLM reserves ~10% of the card for CUDA kernels, the scheduler, and fragmentation headroom.¹ Everything else is figuring out those two terms.

Two terms, two behaviors:

• Model weights are a fixed cost. Load the model, pay it once, it never changes.
• KV cache is a variable cost. It grows with context length and with the number of concurrent requests, and it's the part that quietly blows up your budget.

Term 1: model weights¶

Weights (GB) = Params (billions) × Bytes per parameter

The "billions" cancels the "per-billion-bytes," so the units work out to GB directly. An 8B model at FP16 (2 bytes) is 8 × 2 = 16 GB. That's it.

Bytes per parameter is set by precision, and this is where quantization earns its keep:

How to read precision off a model: check config.json. If torch_dtype is bfloat16 or float16, it's 2 bytes. If the model name says FP8, it's 1 byte. If it says GPTQ or AWQ, or quantization_config shows bits: 4, it's 0.5.

MoE gotcha: for a Mixture-of-Experts model, size by total parameters, not active ones. Qwen3-30B-A3B activates 3B per token but you still have to load all 30B into memory. Use 30, not 3.

So why bother with MoE if it costs the same VRAM? Because MoE saves compute, not memory. All 30B sit in VRAM (the router can pick any expert next token), but each token only runs ~3B worth of math. You get near-30B quality at near-3B speed, which is a great deal when you have the VRAM but are latency- or throughput-bound. It's a bad deal when you're VRAM-constrained: a dense 3B fits where a 30B-A3B never will.

Term 2: the KV cache (the one that bites)¶

Every token you've already processed parks a key and a value vector in memory so the model doesn't recompute them. That's the KV cache, and here's the formula:

KV Cache (GB) = 2 × Layers × Hidden × Context × Batch × Bytes × GQA / 10^9

The factor most people forget is GQA. Modern models share one set of keys/values across several query heads, which shrinks the cache dramatically.⁴ Llama-3.1-8B has 32 attention heads but only 8 KV heads, so GQA = 8 / 32 = 0.25 and the cache is 4× smaller than it looks. If you can't find the heads, leave GQA at 1 (conservative, it overestimates).

Watch it explode with context¶

Same Llama-3.1-8B, FP16, batch 1, just turning the context dial:

The weights didn't move. The cache more than doubled the footprint. Now multiply the cache column by your batch size: 16 concurrent 128K sessions would need ~275 GB of KV cache alone. This is why "it ran fine on my laptop" turns into an OOM the moment real traffic shows up.

Practical lever: if your real requests are short, set a lower max context. A 128K-capable model serving 16K prompts only needs the 16K-sized cache. You don't pay for context you don't use.

Pull the 8 inputs from config.json¶

Two-minute checklist for any Hugging Face model:

• Params (B): from the model name (Llama-3.1-70B → 70; for MoE use the big number)
• Layers: num_hidden_layers
• Hidden: hidden_size
• Context: max_position_embeddings (note: thinking modes can add 20–50K tokens)
• Batch: your call (start at 1)
• Weight precision: torch_dtype / name / quantization_config
• KV precision: match weights unless you're squeezed
• GQA: num_key_value_heads / num_attention_heads

Plug them into the two formulas, add, divide by 0.9. Done.

The card-capacity reality check¶

A "96 GB" card does not give you 96 GB. Two taxes apply:

1. GB vs GiB. Vendors count in decimal GB; the GPU reports binary GiB. That's a ~7% haircut (÷ 1024³ instead of ÷ 10⁹).
2. ECC. Error correction on GDDR memory reserves ~6.25% of capacity.

Usable (GiB) = (Nominal_GB × 10^9 / 1024^3) × (1 − 0.0625)

HBM exception: data-center cards with HBM (H100 80GB, H200 141GB, A100) use dedicated out-of-band ECC, so they don't lose capacity to error correction. For those, usable ≈ nominal in GiB (an 80 GB H100 gives ~74.5 GiB after the binary conversion only). Apply the 6.25% ECC tax only to GDDR cards.

So the Llama-3.1-8B-at-128K example (~36.9 GB) fits comfortably on a single RTX PRO 6000 (~84 GiB usable), with room for a healthy batch. A 236B MoE like DeepSeek-V2.5 at FP8 needs ~267 GB, which is 4 of those cards minimum.⁶

Once the answer is "more than one card," sizing hands off to parallelism — how you actually spread those weights across GPUs (and nodes) is the DP, PP, and TP guide.

Training and fine-tuning are different animals¶

Inference is cheap. Training is where VRAM goes to die, because you also store gradients and optimizer state.

Full training from scratch¶

Static (GB)     = Params × Optimizer multiplier
Activations (GB)= Layers × Hidden × SeqLen × Batch × ActMult / 10^9
Total (GB)      = (Static + Activations) / 0.9

The optimizer multiplier is the killer. Standard mixed-precision AdamW is 16 (2 weights + 2 gradients + 12 optimizer state). So a 7B model needs 7 × 16 = 112 GB of static memory before activations. Add ~9 GB of activations and you're at ~135 GB to train a "small" 7B model.⁷

ActMult is 34 for standard, or 2 with gradient checkpointing (saves ~90% of activation memory, costs ~20% speed).

LoRA / QLoRA fine-tuning¶

This is the sane option. You freeze the base model and train tiny adapters, so there's almost no optimizer state:

Static (GB) = (Params × Base precision) + 1   ← +1 GB covers LoRA rank ~16–32
Activations = Layers × Hidden × SeqLen × Batch × ActMult / 10^9   ← ActMult 14, or 2 with checkpointing
Total       = (Static + Activations) / 0.9

Fine-tuning that same 7B with 4-bit QLoRA, batch 4, 2K sequence: (7 × 0.5 + 1) = 4.5 GB static, ~15 GB activations, ~22 GB total. That fits on a single 24 GB consumer card. The difference between 135 GB and 22 GB is the entire reason QLoRA exists.

Gotchas that wreck the estimate¶

• Vision models add a projector and an image encoder. Add ~1B params to your weight estimate for a VL model.
• Context is input + output combined. A 128K model handed a 128K prompt has zero room left to answer.
• Thinking/reasoning modes silently extend output by 20–50K tokens. Size the cache for the worst case.
• Batch is a multiplier on the cache, not the weights. Doubling concurrency doubles only the KV term.
• KV precision can be dropped independently. If you're 5 GB short, FP8 KV cache (1 byte) halves the cache without touching weight quality.

Summary¶

The whole method on one card:

• Weights = Params × Bytes (FP16=2, FP8=1, INT4=0.5). Fixed cost.
• KV cache = 2 × Layers × Hidden × Context × Batch × Bytes × GQA / 10⁹. Variable cost, scales with context and batch.
• Inference total = (Weights + KV) / 0.9.
• Card usable = nominal minus the GiB conversion, minus 6.25% ECC on GDDR cards (HBM cards skip the ECC tax).
• Fine-tuning with QLoRA, not full training, unless you have 80 GB cards to spare.
• When you're short: drop KV precision, cap context to real demand, or quantize weights, in that order.

Size before you provision. The arithmetic takes two minutes and saves you a multi-thousand-dollar GPU you didn't need, or the OOM you didn't see coming.

Questions or discussion? Connect on LinkedIn, X or reach out via email.

Discussion

Have thoughts on this post? Share them below — questions, corrections, or your own experience are all welcome.