NVIDIA GeForce RTX 5090 (32 GB GDDR7) — Runnable LLMs, Max Context & tok/s Benchmarks

Physical Memory: 32 GB
Default / Tuned Usable: 29.4 / 30.7 GB
Memory Bandwidth: 1,792 GB/s
FP16 Compute: 209.5 TFLOPS
Typical Cloud $/hr / CapEx: $0.79/hr · $2,199
TDP / Native FP8: 575W · FP8 Yes

Load in VRAM Calculator → Generate vLLM / Ollama Config → Compare Self-Hosted GPU vs API →

Interactive Workload Simulator on RTX 5090 32GB

Open-Weight Model Preset (or Custom)

Multi-Device Count (1× to 8× RTX 5090 32GB)

Weight Quantization (bpw)

KV Cache Precision

Context Length (tokens) 32,768 tokens

Memory Breakdown & Bandwidth Ceiling

Fits Comfortably

Model Weights: 79.2% KV: 13.8% Overhead: 7%

Total Required Memory

21.75 GB

Per-Device Allocation: 21.75 / 32 GB

Model Weights

17.22 GB

KV: 3.00 GB

Est. Single-Stream Decode

214.7 tok/s

Est. Batched Throughput: 214.7 tok/s

Max Context Before OOM

132,520

tokens

Key Flag Guardrails Compiled Automatically

Tensor parallelism splits attention heads and MLP matrices across NVLink or PCIe devices, so --tensor-parallel-size must evenly divide query and KV head counts while leaving headroom for NCCL all-reduce buffers and CUDA graph capture.

During single-user autoregressive generation, every output token requires reading active model weights from VRAM into compute cores. Doubling memory bandwidth (for example jumping from 546 GB/s to 1,792 GB/s on GDDR7 or 3,350 GB/s on HBM3) nearly doubles tok/s at identical quantization.

Open-Weight LLM Compatibility Matrix on 1× RTX 5090 32GB (32K Context)

Fit status and estimated single-stream decode tok/s across FP16, Q8_0, and Q4_K_M on a single device
Model & Family	FP16	Q8_0	Q4_K_M	Best Comfortable Quant	Max Context (Q4_K_M)	Decode Speed (Q4_K_M)
DeepSeek-R1 (671B MoE) 671B · MLA	Out of Memory (OOM)	Out of Memory (OOM)	Out of Memory (OOM)	—	—	—
DeepSeek-V3-0324 (671B MoE) 671B · MLA	Out of Memory (OOM)	Out of Memory (OOM)	Out of Memory (OOM)	—	—	—
DeepSeek-R1-Distill-Llama-70B 70.6B · GQA	Out of Memory (OOM)	Out of Memory (OOM)	Out of Memory (OOM)	—	—	—
DeepSeek-R1-Distill-Qwen-32B 32.8B · GQA	Out of Memory (OOM)	Out of Memory (OOM)	Tight Fit (>90% Usable)	Q4_K_M	44,216 tokens	39.4 tok/s
Qwen3-235B-A22B (MoE) 235B · GQA	Out of Memory (OOM)	Out of Memory (OOM)	Out of Memory (OOM)	—	—	—
Qwen3-30B-A3B (MoE) 30.5B · GQA	Out of Memory (OOM)	Out of Memory (OOM)	Fits Comfortably	Q5_K_M	132,520 tokens	214.7 tok/s
Qwen3-32B 32.8B · GQA	Out of Memory (OOM)	Out of Memory (OOM)	Tight Fit (>90% Usable)	Q4_K_M	44,216 tokens	39.4 tok/s
Qwen3-14B 14.8B · GQA	Out of Memory (OOM)	Fits Comfortably	Fits Comfortably	FP8	139,349 tokens	78.2 tok/s
Qwen3-8B 8.2B · GQA	Fits Comfortably	Fits Comfortably	Fits Comfortably	FP16 / BF16	182,781 tokens	114.4 tok/s
QwQ-32B (Reasoning) 32.8B · GQA	Out of Memory (OOM)	Out of Memory (OOM)	Tight Fit (>90% Usable)	Q4_K_M	44,216 tokens	39.4 tok/s
Qwen2.5-Coder-32B-Instruct 32.8B · GQA	Out of Memory (OOM)	Out of Memory (OOM)	Tight Fit (>90% Usable)	Q4_K_M	44,216 tokens	39.4 tok/s
Qwen2.5-72B-Instruct 72.7B · GQA	Out of Memory (OOM)	Out of Memory (OOM)	Out of Memory (OOM)	—	—	—
Llama 4 Maverick (400B / 17B-128E MoE) 402B · GQA	Out of Memory (OOM)	Out of Memory (OOM)	Out of Memory (OOM)	—	—	—
Llama 4 Scout (109B / 17B-16E MoE) 109B · HYBRID_SWA	Out of Memory (OOM)	Out of Memory (OOM)	Out of Memory (OOM)	—	—	—
Llama 3.3 70B Instruct 70.6B · GQA	Out of Memory (OOM)	Out of Memory (OOM)	Out of Memory (OOM)	—	—	—
Llama 3.1 405B Instruct 405B · GQA	Out of Memory (OOM)	Out of Memory (OOM)	Out of Memory (OOM)	—	—	—
Llama 3.1 8B Instruct 8.03B · GQA	Fits Comfortably	Fits Comfortably	Fits Comfortably	FP16 / BF16	206,439 tokens	122.3 tok/s
Gemma 3 27B IT (Multimodal) 27.4B · HYBRID_SWA	Out of Memory (OOM)	Out of Memory (OOM)	Fits Comfortably	Q6_K	29,460 tokens	56.8 tok/s
Gemma 3 12B IT (Multimodal) 12.2B · HYBRID_SWA	Tight Fit (>90% Usable)	Fits Comfortably	Fits Comfortably	FP16 / BF16	62,191 tokens	113.5 tok/s
Mistral Small 3.1 (24B) 24B · GQA	Out of Memory (OOM)	Tight Fit (>90% Usable)	Fits Comfortably	Q6_K	104,285 tokens	56.3 tok/s
Mistral Large 2411 (123B) 123B · GQA	Out of Memory (OOM)	Out of Memory (OOM)	Out of Memory (OOM)	—	—	—
Codestral 22B 22.2B · GQA	Out of Memory (OOM)	Tight Fit (>90% Usable)	Fits Comfortably	Q6_K	79,389 tokens	53.4 tok/s
Devstral Small (24B SWE Agent) 23.6B · GQA	Out of Memory (OOM)	Tight Fit (>90% Usable)	Fits Comfortably	Q6_K	105,810 tokens	57.0 tok/s
GLM-4-32B-0414 32B · GQA	Out of Memory (OOM)	Out of Memory (OOM)	Fits Comfortably	Q5_K_M	193,562 tokens	52.3 tok/s
GLM-4-9B-0414 9.4B · GQA	Fits Comfortably	Fits Comfortably	Fits Comfortably	FP16 / BF16	639,720 tokens	159.2 tok/s
Microsoft Phi-4 (14B) 14.7B · GQA	Out of Memory (OOM)	Fits Comfortably	Fits Comfortably	FP8	111,784 tokens	71.8 tok/s
Microsoft Phi-4-mini-instruct (3.8B) 3.84B · GQA	Fits Comfortably	Fits Comfortably	Fits Comfortably	FP16 / BF16	226,400 tokens	169.3 tok/s
Cohere Command A (111B) 111B · HYBRID_SWA	Out of Memory (OOM)	Out of Memory (OOM)	Out of Memory (OOM)	—	—	—