LLM VRAM, KV Cache & Throughput Calculator

Interactive VRAM, KV Cache & Decode Speed Workbench

Open-Weight Model Preset (or Custom)

Weight Quantization (bpw)

KV Cache Precision

Target GPU / Unified Memory Tier

Devices (TP)

Context Length (tokens) 32,768 tokens

Concurrent Sequences (Batch) 1

Inspect or override raw transformer dimensions (layers, KV heads, MLA rank, MoE experts)

Topology & Attention

Attention Mechanism

Total Params (B)

Active Params / Token (B)

Hidden Layers

Hidden Size

Query Heads

KV Heads

Head Dim

MLA kv_lora_rank

SWA Window

Memory Breakdown & Bandwidth Ceiling

Tight Fit (>90% Usable)

Model Weights: 65.5% KV Cache: 28.3% Runtime & TP Overhead: 6.2%

Total Required Memory

28.28 GB

88.4% of 32 GB

Per-Device Allocation

28.28 GB

Usable: 29.4 GB

Model Weights

18.52 GB

Runtime & TP Overhead: 1.76 GB

KV Cache (FP16)

8.00 GB

MHA FP16: 64.00 GB

Est. Single-Stream Decode

39.4 tok/s

Est. Batched Throughput: 39.4 tok/s

Max Context Before OOM

44,216

tokens

Architecture KV compression saves 87.5% vs naive MHA FP16 (64.00 GB).

Generate vLLM / Ollama Config → Compare Self-Hosted GPU vs API → Compare Live API Token Rates on DevToken →

Jump to Specialized GPU Serving Workbenches

GGUF & config.json Inspector

Inspect multi-gigabyte .gguf model files and Transformer config.json metadata locally in 0ms via File.slice and DataView—zero file uploads, instant VRAM handoff.

File.slice(0..16MiB) · 0 Upload

Serving Config Generator

Compile OOM-safe vLLM CLI flags, SGLang server commands, Ollama Modelfiles, Docker Compose GPU stacks, and Kubernetes Deployment YAML from real LLM VRAM math.

vLLM · SGLang · Ollama · K8s

GPU vs Cloud API Break-Even

Model monthly GPU rental or local workstation costs against Cloud LLM API token bills, prompt-caching discounts, and physical tok/s throughput saturation.

FinOps 30-Day ROI · DevToken

Open-Weight LLM VRAM Quick Reference (28 Models @ Q4_K_M & FP16)

Showing 28 of 28 models

Verified open-weight LLM architectures with resident parameters, attention mechanism, and baseline VRAM requirements
Model & Family	Topology & Attention	Resident / Active	FP16 (8K ctx)	Q4_K_M (32K ctx)	Minimum Practical Tier	Deep Dive
DeepSeek-R1 (671B MoE) DeepSeek · MIT	MOE MLA	671B / 37B	1289.2 GB	393.7 GB	8× NVIDIA H100 80GB	VRAM Guide →
DeepSeek-V3-0324 (671B MoE) DeepSeek · MIT	MOE MLA	671B / 37B	1289.2 GB	393.7 GB	8× NVIDIA H100 80GB	VRAM Guide →
DeepSeek-R1-Distill-Llama-70B DeepSeek · MIT	DENSE GQA	70.6B	139.3 GB	52.5 GB	NVIDIA A100 80GB	VRAM Guide →
DeepSeek-R1-Distill-Qwen-32B DeepSeek · MIT	DENSE GQA	32.8B	66.1 GB	28.3 GB	RTX 5090 32GB	VRAM Guide →
Qwen3-235B-A22B (MoE) Qwen · Apache-2.0	MOE GQA	235B / 22B	453.5 GB	143.7 GB	Mac Studio 192GB	VRAM Guide →
Qwen3-30B-A3B (MoE) Qwen · Apache-2.0	MOE GQA	30.5B / 3.3B	60.3 GB	21.8 GB	RTX 3090 24GB	VRAM Guide →
Qwen3-32B Qwen · Apache-2.0	DENSE GQA	32.8B	66.1 GB	28.3 GB	RTX 5090 32GB	VRAM Guide →
Qwen3-14B Qwen · Apache-2.0	DENSE GQA	14.8B	30.9 GB	14.8 GB	RTX 4060 Ti 16GB	VRAM Guide →
Qwen3-8B Qwen · Apache-2.0	DENSE GQA	8.2B	18.0 GB	10.4 GB	RTX 3060 12GB	VRAM Guide →
QwQ-32B (Reasoning) Qwen · Apache-2.0	DENSE GQA	32.8B	66.1 GB	28.3 GB	RTX 5090 32GB	VRAM Guide →
Qwen2.5-Coder-32B-Instruct Qwen · Apache-2.0	DENSE GQA	32.8B	66.1 GB	28.3 GB	RTX 5090 32GB	VRAM Guide →
Qwen2.5-72B-Instruct Qwen · Qwen License	DENSE GQA	72.7B	143.4 GB	53.7 GB	NVIDIA A100 80GB	VRAM Guide →
Llama 4 Maverick (400B / 17B-128E MoE) Meta Llama · Llama 4 Community	MOE GQA	402B / 17.2B	774.0 GB	241.0 GB	Mac Studio 512GB	VRAM Guide →
Llama 4 Scout (109B / 17B-16E MoE) Meta Llama · Llama 4 Community	MOE HYBRID_SWA	109B / 17.2B	211.8 GB	67.2 GB	NVIDIA A100 80GB	VRAM Guide →
Llama 3.3 70B Instruct Meta Llama · Llama 3.3 Community	DENSE GQA	70.6B	139.3 GB	52.5 GB	NVIDIA A100 80GB	VRAM Guide →
Llama 3.1 405B Instruct Meta Llama · Llama 3.1 Community	DENSE GQA	405B	782.8 GB	253.2 GB	Mac Studio 512GB	VRAM Guide →
Llama 3.1 8B Instruct Meta Llama · Llama 3.1 Community	DENSE GQA	8.03B	17.6 GB	9.8 GB	RTX 3060 12GB	VRAM Guide →
Gemma 3 27B IT (Multimodal) Google Gemma · Gemma Terms	DENSE HYBRID_SWA	27.4B	54.8 GB	20.1 GB	RTX 3090 24GB	VRAM Guide →
Gemma 3 12B IT (Multimodal) Google Gemma · Gemma Terms	DENSE HYBRID_SWA	12.2B	25.3 GB	10.5 GB	RTX 3060 12GB	VRAM Guide →
Mistral Small 3.1 (24B) Mistral · Apache-2.0	DENSE GQA	24B	48.5 GB	20.2 GB	RTX 3090 24GB	VRAM Guide →
Mistral Large 2411 (123B) Mistral · Mistral Research	DENSE GQA	123B	240.4 GB	84.2 GB	RTX PRO 6000 96GB	VRAM Guide →
Codestral 22B Mistral · Mistral Non-Production	DENSE GQA	22.2B	45.6 GB	21.2 GB	RTX 3090 24GB	VRAM Guide →
Devstral Small (24B SWE Agent) Mistral · Apache-2.0	DENSE GQA	23.6B	47.7 GB	19.9 GB	RTX 3090 24GB	VRAM Guide →
GLM-4-32B-0414 Zhipu GLM · MIT	DENSE GQA	32B	63.1 GB	21.8 GB	RTX 3090 24GB	VRAM Guide →
GLM-4-9B-0414 Zhipu GLM · MIT	DENSE GQA	9.4B	19.5 GB	7.9 GB	RTX 3060 12GB	VRAM Guide →
Microsoft Phi-4 (14B) Microsoft Phi · MIT	DENSE GQA	14.7B	31.0 GB	16.0 GB	RTX 3090 24GB	VRAM Guide →
Microsoft Phi-4-mini-instruct (3.8B) Microsoft Phi · MIT	DENSE GQA	3.84B	9.4 GB	7.3 GB	RTX 3060 12GB	VRAM Guide →
Cohere Command A (111B) Cohere · CC-BY-NC-4.0	DENSE HYBRID_SWA	111B	215.8 GB	68.9 GB	NVIDIA A100 80GB	VRAM Guide →

Flagship GPU & Apple Silicon Capacity Guides

RTX 3090 24GB

Consumer GPU · 24 GB

936 GB/s · $0.29/hr

RTX 4090 24GB

Consumer GPU · 24 GB

1,008 GB/s · $0.44/hr

RTX 5090 32GB

Consumer GPU · 32 GB

1,792 GB/s · $0.79/hr

Apple M-Max 64GB

Apple Silicon Unified · 64 GB

546 GB/s · $0.22/hr

Mac Studio 128GB

Apple Silicon Unified · 128 GB

546 GB/s · $0.32/hr

Mac Studio 192GB

Apple Silicon Unified · 192 GB

800 GB/s · $0.48/hr

NVIDIA A100 80GB

Datacenter GPU · 80 GB

2,039 GB/s · $1.49/hr

NVIDIA H100 80GB

Datacenter GPU · 80 GB

3,350 GB/s · $2.39/hr

Why Naive "2 GB per Billion Parameters" Formulas Fail in Production

Back-of-the-envelope rules ignore three dominant variables in modern open-weight inference: GGUF super-block quantization scales (where Q4_K_M averages 4.85 bits/weight rather than 4.00), KV cache growth across long context windows and concurrent continuous-batching slots, and Mixture-of-Experts routing where all 671B or 235B parameters sit in VRAM while only 37B or 22B stream per generated token.

VRAMLab evaluates each model from its verified transformer geometry—branching explicitly across Multi-Head Attention (MHA), Grouped-Query Attention (GQA), DeepSeek Multi-head Latent Attention (MLA), and hybrid local/global sliding-window stacks—so capacity plans match real vLLM, SGLang, Ollama, and llama.cpp allocations.

Frequently Asked Questions

How does DeepSeek MLA reduce KV cache VRAM compared with standard GQA?

Multi-head Latent Attention projects keys and values into a shared low-rank latent vector (such as kv_lora_rank = 512 plus qk_rope_head_dim = 64) instead of caching 2 × H_kv × head_dim tensors per layer, cutting long-context KV cache memory by over 90% relative to uncompressed MHA.

Why do Mixture-of-Experts (MoE) models require high VRAM yet decode quickly?

Every routed expert must stay resident in GPU or unified memory to avoid PCIe/SSD thrashing, so VRAM scales with total parameters (e.g., 235B for Qwen3-235B-A22B). Decode tok/s, however, is memory-bandwidth bound on only the active parameters per token (22B), yielding dense-32B-like speed once fitted.

Why does macOS trigger OOM on Apple Silicon before 100% of unified RAM is used?

macOS reserves unified memory for kernel and desktop compositing via iogpu.wired_limit_mb (~75% of physical RAM by default). Models between 75% and 88% of RAM require raising iogpu.wired_limit_mb via sysctl prior to launching Ollama or llama.cpp.

Where do VRAMLab architecture dimensions come from?

Our 12-hour automated architecture verification pipeline validates numeric transformer configuration fields across official open-weight releases and compiles them into static HTML—requiring zero runtime API calls or user telemetry.

Architecture-Aware LLM VRAM & KV Cache Calculator