GPU & Apple Silicon LLM VRAM & Bandwidth Tiers

Memory Bandwidth Is the Primary Bottleneck for Local LLM Decode

During single-user autoregressive generation, every output token requires reading active model weights from VRAM into compute cores. Doubling memory bandwidth (for example jumping from 546 GB/s to 1,792 GB/s on GDDR7 or 3,350 GB/s on HBM3) nearly doubles tok/s at identical quantization.

On macOS Apple Silicon systems, default GPU wired allocation stops near 75% of physical unified RAM unless overridden with sysctl iogpu.wired_limit_mb (modeled up to an 88% safe ceiling that preserves OS stability). Autoregressive decode speed is bounded by effective external memory bandwidth divided by active parameter bytes plus active KV cache bytes read per step, verified every 12 hours by the VRAMLab automated architecture verification pipeline.

Tensor parallelism splits attention heads and MLP matrices across NVLink or PCIe devices, so --tensor-parallel-size must evenly divide query and KV head counts while leaving headroom for NCCL all-reduce buffers and CUDA graph capture.