NVIDIA GeForce RTX 3090 (24 GB GDDR6X) — Runnable LLMs, Max Context & tok/s Benchmarks
NVIDIA GeForce RTX 3090 (24 GB GDDR6X) provides 24 GB of memory (22.1 GB default usable ceiling) backed by 936 GB/s of external bandwidth and 71 TFLOPS FP16 compute (~$0.29/hr typical cloud rate, 350W TDP).
- Physical Memory
- 24 GB
- Default / Tuned Usable
- 22.1 / 23.0 GB
- Memory Bandwidth
- 936 GB/s
- FP16 Compute
- 71 TFLOPS
- Typical Cloud $/hr / CapEx
- $0.29/hr · $780
- TDP / Native FP8
- 350W · FP16/INT4
Interactive Workload Simulator on RTX 3090 24GB
Key Flag Guardrails Compiled Automatically
Tensor parallelism splits attention heads and MLP matrices across NVLink or PCIe devices, so --tensor-parallel-size must evenly divide query and KV head counts while leaving headroom for NCCL all-reduce buffers and CUDA graph capture.
During single-user autoregressive generation, every output token requires reading active model weights from VRAM into compute cores. Doubling memory bandwidth (for example jumping from 546 GB/s to 1,792 GB/s on GDDR7 or 3,350 GB/s on HBM3) nearly doubles tok/s at identical quantization.
Open-Weight LLM Compatibility Matrix on 1× RTX 3090 24GB (32K Context)
| Model & Family | FP16 | Q8_0 | Q4_K_M | Best Comfortable Quant | Max Context (Q4_K_M) | Decode Speed (Q4_K_M) |
|---|---|---|---|---|---|---|
|
DeepSeek-R1 (671B MoE)
671B · MLA
|
Out of Memory (OOM) | Out of Memory (OOM) | Out of Memory (OOM) | — | — | — |
|
DeepSeek-V3-0324 (671B MoE)
671B · MLA
|
Out of Memory (OOM) | Out of Memory (OOM) | Out of Memory (OOM) | — | — | — |
|
DeepSeek-R1-Distill-Llama-70B
70.6B · GQA
|
Out of Memory (OOM) | Out of Memory (OOM) | Out of Memory (OOM) | — | — | — |
|
DeepSeek-R1-Distill-Qwen-32B
32.8B · GQA
|
Out of Memory (OOM) | Out of Memory (OOM) | Out of Memory (OOM) | — | 12,759 tokens | — |
|
Qwen3-235B-A22B (MoE)
235B · GQA
|
Out of Memory (OOM) | Out of Memory (OOM) | Out of Memory (OOM) | — | — | — |
|
Qwen3-30B-A3B (MoE)
30.5B · GQA
|
Out of Memory (OOM) | Out of Memory (OOM) | Tight Fit (>90% Usable) | IQ4_XS | 48,634 tokens | 112.1 tok/s |
|
Qwen3-32B
32.8B · GQA
|
Out of Memory (OOM) | Out of Memory (OOM) | Out of Memory (OOM) | — | 12,759 tokens | — |
|
Qwen3-14B
14.8B · GQA
|
Out of Memory (OOM) | Tight Fit (>90% Usable) | Fits Comfortably | Q8_0 | 89,017 tokens | 40.8 tok/s |
|
Qwen3-8B
8.2B · GQA
|
Tight Fit (>90% Usable) | Fits Comfortably | Fits Comfortably | FP16 / BF16 | 126,857 tokens | 59.7 tok/s |
|
QwQ-32B (Reasoning)
32.8B · GQA
|
Out of Memory (OOM) | Out of Memory (OOM) | Out of Memory (OOM) | — | 12,759 tokens | — |
|
Qwen2.5-Coder-32B-Instruct
32.8B · GQA
|
Out of Memory (OOM) | Out of Memory (OOM) | Out of Memory (OOM) | — | 12,759 tokens | — |
|
Qwen2.5-72B-Instruct
72.7B · GQA
|
Out of Memory (OOM) | Out of Memory (OOM) | Out of Memory (OOM) | — | — | — |
|
Llama 4 Maverick (400B / 17B-128E MoE)
402B · GQA
|
Out of Memory (OOM) | Out of Memory (OOM) | Out of Memory (OOM) | — | — | — |
|
Llama 4 Scout (109B / 17B-16E MoE)
109B · HYBRID_SWA
|
Out of Memory (OOM) | Out of Memory (OOM) | Out of Memory (OOM) | — | — | — |
|
Llama 3.3 70B Instruct
70.6B · GQA
|
Out of Memory (OOM) | Out of Memory (OOM) | Out of Memory (OOM) | — | — | — |
|
Llama 3.1 405B Instruct
405B · GQA
|
Out of Memory (OOM) | Out of Memory (OOM) | Out of Memory (OOM) | — | — | — |
|
Llama 3.1 8B Instruct
8.03B · GQA
|
Tight Fit (>90% Usable) | Fits Comfortably | Fits Comfortably | FP16 / BF16 | 143,524 tokens | 63.9 tok/s |
|
Gemma 3 27B IT (Multimodal)
27.4B · HYBRID_SWA
|
Out of Memory (OOM) | Out of Memory (OOM) | Tight Fit (>90% Usable) | Q4_K_M | 13,224 tokens | 29.7 tok/s |
|
Gemma 3 12B IT (Multimodal)
12.2B · HYBRID_SWA
|
Out of Memory (OOM) | Fits Comfortably | Fits Comfortably | Q8_0 | 41,219 tokens | 59.3 tok/s |
|
Mistral Small 3.1 (24B)
24B · GQA
|
Out of Memory (OOM) | Out of Memory (OOM) | Tight Fit (>90% Usable) | Q4_K_M | 53,954 tokens | 29.4 tok/s |
|
Mistral Large 2411 (123B)
123B · GQA
|
Out of Memory (OOM) | Out of Memory (OOM) | Out of Memory (OOM) | — | — | — |
|
Codestral 22B
22.2B · GQA
|
Out of Memory (OOM) | Out of Memory (OOM) | Tight Fit (>90% Usable) | Q4_K_M | 43,438 tokens | 27.9 tok/s |
|
Devstral Small (24B SWE Agent)
23.6B · GQA
|
Out of Memory (OOM) | Out of Memory (OOM) | Tight Fit (>90% Usable) | Q5_K_M | 55,478 tokens | 29.8 tok/s |
|
GLM-4-32B-0414
32B · GQA
|
Out of Memory (OOM) | Out of Memory (OOM) | Tight Fit (>90% Usable) | IQ4_XS | 61,544 tokens | 27.3 tok/s |
|
GLM-4-9B-0414
9.4B · GQA
|
Tight Fit (>90% Usable) | Fits Comfortably | Fits Comfortably | FP16 / BF16 | 438,394 tokens | 83.2 tok/s |
|
Microsoft Phi-4 (14B)
14.7B · GQA
|
Out of Memory (OOM) | Tight Fit (>90% Usable) | Fits Comfortably | Q8_0 | 71,519 tokens | 37.5 tok/s |
|
Microsoft Phi-4-mini-instruct (3.8B)
3.84B · GQA
|
Fits Comfortably | Fits Comfortably | Fits Comfortably | FP16 / BF16 | 163,486 tokens | 88.4 tok/s |
|
Cohere Command A (111B)
111B · HYBRID_SWA
|
Out of Memory (OOM) | Out of Memory (OOM) | Out of Memory (OOM) | — | — | — |
Compare Other GPU & Apple Silicon Capacity Tiers
RTX 4090 24GB
24 GB · 1,008 GB/s
RTX 5090 32GB
32 GB · 1,792 GB/s
Apple M-Max 64GB
64 GB · 546 GB/s
Mac Studio 128GB
128 GB · 546 GB/s
Mac Studio 192GB
192 GB · 800 GB/s
NVIDIA A100 80GB
80 GB · 2,039 GB/s
NVIDIA H100 80GB
80 GB · 3,350 GB/s