NVIDIA A100 SXM4 (80 GB HBM2e) — Runnable LLMs, Max Context & tok/s Benchmarks

Physical Memory: 80 GB
Default / Tuned Usable: 73.6 / 76.8 GB
Memory Bandwidth: 2,039 GB/s
FP16 Compute: 312 TFLOPS
Typical Cloud $/hr / CapEx: $1.49/hr · $14,500
TDP / Native FP8: 400W · FP16/INT4

Load in VRAM Calculator → Generate vLLM / Ollama Config → Compare Self-Hosted GPU vs API →

Interactive Workload Simulator on NVIDIA A100 80GB

Open-Weight Model Preset (or Custom)

Multi-Device Count (1× to 8× NVIDIA A100 80GB)

Weight Quantization (bpw)

KV Cache Precision

Context Length (tokens) 32,768 tokens

Memory Breakdown & Bandwidth Ceiling

Fits Comfortably

Model Weights: 76% KV: 19.1% Overhead: 4.9%

Total Required Memory

52.45 GB

Per-Device Allocation: 52.45 / 80 GB

Model Weights

39.86 GB

KV: 10.00 GB

Est. Single-Stream Decode

25.2 tok/s

Est. Batched Throughput: 25.2 tok/s

Max Context Before OOM

114,335

tokens

Key Flag Guardrails Compiled Automatically

Tensor parallelism splits attention heads and MLP matrices across NVLink or PCIe devices, so --tensor-parallel-size must evenly divide query and KV head counts while leaving headroom for NCCL all-reduce buffers and CUDA graph capture.

During single-user autoregressive generation, every output token requires reading active model weights from VRAM into compute cores. Doubling memory bandwidth (for example jumping from 546 GB/s to 1,792 GB/s on GDDR7 or 3,350 GB/s on HBM3) nearly doubles tok/s at identical quantization.

Open-Weight LLM Compatibility Matrix on 1× NVIDIA A100 80GB (32K Context)

Fit status and estimated single-stream decode tok/s across FP16, Q8_0, and Q4_K_M on a single device
Model & Family	FP16	Q8_0	Q4_K_M	Best Comfortable Quant	Max Context (Q4_K_M)	Decode Speed (Q4_K_M)
DeepSeek-R1 (671B MoE) 671B · MLA	Out of Memory (OOM)	Out of Memory (OOM)	Out of Memory (OOM)	—	—	—
DeepSeek-V3-0324 (671B MoE) 671B · MLA	Out of Memory (OOM)	Out of Memory (OOM)	Out of Memory (OOM)	—	—	—
DeepSeek-R1-Distill-Llama-70B 70.6B · GQA	Out of Memory (OOM)	Out of Memory (OOM)	Fits Comfortably	Q6_K	114,335 tokens	25.2 tok/s
DeepSeek-R1-Distill-Qwen-32B 32.8B · GQA	Tight Fit (>90% Usable)	Fits Comfortably	Fits Comfortably	Q8_0	232,960 tokens	47.4 tok/s
Qwen3-235B-A22B (MoE) 235B · GQA	Out of Memory (OOM)	Out of Memory (OOM)	Out of Memory (OOM)	—	—	—
Qwen3-30B-A3B (MoE) 30.5B · GQA	Fits Comfortably	Fits Comfortably	Fits Comfortably	FP16 / BF16	635,837 tokens	258.7 tok/s
Qwen3-32B 32.8B · GQA	Tight Fit (>90% Usable)	Fits Comfortably	Fits Comfortably	Q8_0	232,960 tokens	47.4 tok/s
Qwen3-14B 14.8B · GQA	Fits Comfortably	Fits Comfortably	Fits Comfortably	FP16 / BF16	441,339 tokens	94.2 tok/s
Qwen3-8B 8.2B · GQA	Fits Comfortably	Fits Comfortably	Fits Comfortably	FP16 / BF16	518,326 tokens	137.8 tok/s
QwQ-32B (Reasoning) 32.8B · GQA	Tight Fit (>90% Usable)	Fits Comfortably	Fits Comfortably	Q8_0	232,960 tokens	47.4 tok/s
Qwen2.5-Coder-32B-Instruct 32.8B · GQA	Tight Fit (>90% Usable)	Fits Comfortably	Fits Comfortably	Q8_0	232,960 tokens	47.4 tok/s
Qwen2.5-72B-Instruct 72.7B · GQA	Out of Memory (OOM)	Out of Memory (OOM)	Fits Comfortably	Q6_K	110,333 tokens	24.6 tok/s
Llama 4 Maverick (400B / 17B-128E MoE) 402B · GQA	Out of Memory (OOM)	Out of Memory (OOM)	Out of Memory (OOM)	—	—	—
Llama 4 Scout (109B / 17B-16E MoE) 109B · HYBRID_SWA	Out of Memory (OOM)	Out of Memory (OOM)	Tight Fit (>90% Usable)	IQ4_XS	68,598 tokens	102.0 tok/s
Llama 3.3 70B Instruct 70.6B · GQA	Out of Memory (OOM)	Out of Memory (OOM)	Fits Comfortably	Q6_K	114,335 tokens	25.2 tok/s
Llama 3.1 405B Instruct 405B · GQA	Out of Memory (OOM)	Out of Memory (OOM)	Out of Memory (OOM)	—	—	—
Llama 3.1 8B Instruct 8.03B · GQA	Fits Comfortably	Fits Comfortably	Fits Comfortably	FP16 / BF16	583,926 tokens	147.4 tok/s
Gemma 3 27B IT (Multimodal) 27.4B · HYBRID_SWA	Fits Comfortably	Fits Comfortably	Fits Comfortably	FP16 / BF16	126,876 tokens	68.4 tok/s
Gemma 3 12B IT (Multimodal) 12.2B · HYBRID_SWA	Fits Comfortably	Fits Comfortably	Fits Comfortably	FP16 / BF16	188,020 tokens	136.7 tok/s
Mistral Small 3.1 (24B) 24B · GQA	Fits Comfortably	Fits Comfortably	Fits Comfortably	FP16 / BF16	406,275 tokens	67.8 tok/s
Mistral Large 2411 (123B) 123B · GQA	Out of Memory (OOM)	Out of Memory (OOM)	Out of Memory (OOM)	—	13,163 tokens	—
Codestral 22B 22.2B · GQA	Fits Comfortably	Fits Comfortably	Fits Comfortably	FP16 / BF16	295,097 tokens	64.4 tok/s
Devstral Small (24B SWE Agent) 23.6B · GQA	Fits Comfortably	Fits Comfortably	Fits Comfortably	FP16 / BF16	407,800 tokens	68.6 tok/s
GLM-4-32B-0414 32B · GQA	Fits Comfortably	Fits Comfortably	Fits Comfortably	FP16 / BF16	985,666 tokens	63.0 tok/s
GLM-4-9B-0414 9.4B · GQA	Fits Comfortably	Fits Comfortably	Fits Comfortably	FP16 / BF16	1,048,576 tokens	191.8 tok/s
Microsoft Phi-4 (14B) 14.7B · GQA	Fits Comfortably	Fits Comfortably	Fits Comfortably	FP16 / BF16	353,376 tokens	86.5 tok/s
Microsoft Phi-4-mini-instruct (3.8B) 3.84B · GQA	Fits Comfortably	Fits Comfortably	Fits Comfortably	FP16 / BF16	603,888 tokens	203.9 tok/s
Cohere Command A (111B) 111B · HYBRID_SWA	Out of Memory (OOM)	Out of Memory (OOM)	Tight Fit (>90% Usable)	IQ4_XS	46,684 tokens	19.2 tok/s