Architecture-Aware LLM Memory & Bandwidth Methodology
VRAMLab models LLM inference memory from first-principles tensor dimensions rather than static multiplier heuristics, running 100% client-side on zero-cost static edge assets.
1. Weight Quantization & Effective Bits-per-Weight (bpw)
Quantized formats store per-block scale factors and zero-points alongside packed integer weights. A 4-bit GGUF Q4_K_M checkpoint averages 4.85 effective bits per parameter (not 4.00), while Q8_0 averages 8.50 bpw. For Mixture-of-Experts models, all routed and shared experts remain resident in memory to prevent token-by-token host paging.
| Precision / Format | Effective bpw | 32B Weights | 70B Weights | vLLM / Ollama / GGUF |
|---|---|---|---|---|
| FP16 / BF16 (16-bit Native) | 16.00 bpw | 61.09 GB | 131.50 GB | native · fp16 · F16 / BF16 |
| FP8 E4M3 (Hopper / Ada / Blackwell) | 8.50 bpw | 32.46 GB | 69.86 GB | fp8 · fp8 · FP8_E4M3 |
| INT8 SmoothQuant / bitsandbytes | 8.50 bpw | 32.46 GB | 69.86 GB | bitsandbytes · q8_0 · INT8 |
| GGUF Q8_0 (8.50 bpw) | 8.50 bpw | 32.46 GB | 69.86 GB | gguf · q8_0 · Q8_0 |
| GGUF Q6_K (6.56 bpw) | 6.56 bpw | 25.05 GB | 53.92 GB | gguf · q6_K · Q6_K |
| GGUF Q5_K_M (5.69 bpw) | 5.69 bpw | 21.73 GB | 46.77 GB | gguf · q5_K_M · Q5_K_M |
| GGUF Q4_K_M (4.85 bpw — Recommended) | 4.85 bpw | 18.52 GB | 39.86 GB | gguf · q4_K_M · Q4_K_M |
| AWQ / GPTQ INT4 (4.50 bpw) | 4.50 bpw | 17.18 GB | 36.99 GB | awq · q4_K_M · AWQ_4BIT |
| GGUF IQ4_XS (4.25 bpw Importance Matrix) | 4.25 bpw | 16.23 GB | 34.93 GB | gguf · iq4_XS · IQ4_XS |
| GGUF Q3_K_M (3.91 bpw) | 3.91 bpw | 14.93 GB | 32.14 GB | gguf · q3_K_M · Q3_K_M |
| GGUF Q2_K (3.35 bpw Extreme) | 3.35 bpw | 12.79 GB | 27.53 GB | gguf · q2_K · Q2_K |
2. Attention-Specific KV Cache Equations (MHA, GQA, MLA & Hybrid SWA)
Standard Grouped-Query Attention allocates 2 × L × H_kv × d_head × S × B × bytes_kv across layers. DeepSeek Multi-head Latent Attention (MLA) compresses keys and values into L × (d_kv_lora + d_qk_rope) × S × B × bytes_kv, slashing long-context memory by over 90%. Hybrid sliding-window models cap local-attention layers at min(S, window_size) while scaling global layers linearly with S.
- Multi-Head Attention (MHA) / Grouped-Query Attention (GQA)
- 2 × L × H_kv × d_head × S × B × bytes_kv
- Multi-head Latent Attention (MLA)
- L × (kv_lora_rank + qk_rope_dim) × S × B × bytes_kv
- Hybrid Sliding-Window (SWA)
- (L_global × S + L_local × min(S, W)) × 2 × H_kv × d_head × B × bytes_kv
- Mixture-of-Experts (MoE)
- Weights ∝ P_total · Decode tok/s ∝ BW / P_active
3. Apple Silicon Unified Memory & Decode Bandwidth Upper Bound
On macOS Apple Silicon systems, default GPU wired allocation stops near 75% of physical unified RAM unless overridden with sysctl iogpu.wired_limit_mb (modeled up to an 88% safe ceiling that preserves OS stability). Autoregressive decode speed is bounded by effective external memory bandwidth divided by active parameter bytes plus active KV cache bytes read per step, verified every 12 hours by the VRAMLab automated architecture verification pipeline.
| Accelerator / System | Physical Memory | Default Usable | Tuned Ceiling | Memory Bandwidth |
|---|---|---|---|---|
| NVIDIA GeForce RTX 3090 (24 GB GDDR6X) | 24 GB | 22.1 GB (92%) | 23.0 GB (96%) | 936 GB/s |
| NVIDIA GeForce RTX 4090 (24 GB GDDR6X) | 24 GB | 22.1 GB (92%) | 23.0 GB (96%) | 1,008 GB/s |
| NVIDIA GeForce RTX 5090 (32 GB GDDR7) | 32 GB | 29.4 GB (92%) | 30.7 GB (96%) | 1,792 GB/s |
| Apple Silicon M-Series Max (64 GB Unified Memory) | 64 GB | 48.0 GB (75%) | 56.3 GB (88% sysctl) | 546 GB/s |
| Apple Mac Studio M4 Max (128 GB Unified Memory) | 128 GB | 96.0 GB (75%) | 112.6 GB (88% sysctl) | 546 GB/s |
| Apple Mac Studio M2/M3 Ultra (192 GB Unified Memory) | 192 GB | 144.0 GB (75%) | 169.0 GB (88% sysctl) | 800 GB/s |
| NVIDIA A100 SXM4 (80 GB HBM2e) | 80 GB | 73.6 GB (92%) | 76.8 GB (96%) | 2,039 GB/s |
| NVIDIA H100 SXM5 (80 GB HBM3) | 80 GB | 73.6 GB (92%) | 76.8 GB (96%) | 3,350 GB/s |
4. Connected CodeAP Studio Engineering Workbenches
Pair capacity planning on VRAMLab with live 460+ LLM API pricing & 429 rate-limit modeling on DevToken, 7-ecosystem deprecation tracking on LLMEOL, and PDF/DOCX/HTML-to-Markdown token compression on ToMarkdown.
Frequently Asked Questions
How does DeepSeek MLA reduce KV cache VRAM compared with standard GQA?
Multi-head Latent Attention projects keys and values into a shared low-rank latent vector (such as kv_lora_rank = 512 plus qk_rope_head_dim = 64) instead of caching 2 × H_kv × head_dim tensors per layer, cutting long-context KV cache memory by over 90% relative to uncompressed MHA.
Why do Mixture-of-Experts (MoE) models require high VRAM yet decode quickly?
Every routed expert must stay resident in GPU or unified memory to avoid PCIe/SSD thrashing, so VRAM scales with total parameters (e.g., 235B for Qwen3-235B-A22B). Decode tok/s, however, is memory-bandwidth bound on only the active parameters per token (22B), yielding dense-32B-like speed once fitted.
Why does macOS trigger OOM on Apple Silicon before 100% of unified RAM is used?
macOS reserves unified memory for kernel and desktop compositing via iogpu.wired_limit_mb (~75% of physical RAM by default). Models between 75% and 88% of RAM require raising iogpu.wired_limit_mb via sysctl prior to launching Ollama or llama.cpp.
Where do VRAMLab architecture dimensions come from?
Our 12-hour automated architecture verification pipeline validates numeric transformer configuration fields across official open-weight releases and compiles them into static HTML—requiring zero runtime API calls or user telemetry.