Quick answer: For most local LLM users, 16GB VRAM is the sweet spot. It handles all 7B models and most 13B models quantized. If you want to run 34B+ models, you need 24GB or more.
NVIDIA GeForce RTX 4060 Ti 16GB
16GB GDDR616GB VRAM at $425 — handles all 7B models and most 13B models. The most cost-effective entry into local LLM inference.
Affiliate links — we may earn a commission at no extra cost to you. Amazon ships globally; Shopee SG covers Singapore & ASEAN.
VRAM capacity memory bandwidth Specs are manufacturer figures. Bar lengths are scaled independently per metric.
The VRAM rule of thumb
A rough formula for estimating VRAM needs:
VRAM needed = (Parameters in billions x Bytes per parameter) + 2GB overhead
At FP16 (2 bytes per param): a 7B model needs ~16GB. But with quantization, you can cut that dramatically.
You can sanity-check any model-and-quant combo right here:
VRAM requirements by model size
7B parameter models (Llama 3 8B, Mistral 7B, Gemma 7B)
| Quantization | Model Size | VRAM Needed | Minimum GPU |
|---|---|---|---|
| FP16 | ~14GB | 16GB | RTX 4060 Ti 16GB |
| Q8 | ~7.5GB | 10GB | RTX 3060 12GB |
| Q6_K | ~5.5GB | 8GB | RTX 4060 8GB |
| Q4_K_M | ~4.5GB | 6GB | RTX 3060 12GB |
| Q3_K_M | ~3.5GB | 6GB | Any 6GB GPU |
Verdict: Any modern GPU with 8GB+ VRAM handles 7B models comfortably. 16GB gives you room for full precision and longer contexts. For Llama 3 8B specifically, see how much VRAM Llama 3 8B actually uses across all quantization levels.
13B parameter models (Llama 2 13B, CodeLlama 13B)
| Quantization | Model Size | VRAM Needed | Minimum GPU |
|---|---|---|---|
| FP16 | ~26GB | 28GB | RTX 5090 32GB |
| Q8 | ~13GB | 16GB | RTX 4060 Ti 16GB |
| Q6_K | ~10GB | 12GB | RTX 3060 12GB |
| Q4_K_M | ~7.5GB | 10GB | RTX 3060 12GB |
| Q3_K_M | ~6GB | 8GB | RTX 4060 8GB |
Verdict: 12-16GB VRAM is ideal for 13B models. The RTX 4060 Ti 16GB or RTX 4070 handles these well.
Check RTX 4060 Ti 16GB Price→Buy on Shopee SG→34B parameter models (CodeLlama 34B, Yi 34B)
| Quantization | Model Size | VRAM Needed | Minimum GPU |
|---|---|---|---|
| FP16 | ~68GB | 70GB+ | Multi-GPU required |
| Q8 | ~34GB | 36GB+ | A6000 48GB |
| Q6_K | ~26GB | 28GB | RTX 5090 32GB |
| Q4_K_M | ~20GB | 22GB | RTX 4090 24GB |
| Q3_K_M | ~15GB | 18GB | RTX 4070 Ti Super 16GB (tight) |
Verdict: 24GB is the minimum for comfortable 34B use. The RTX 4090 is the go-to card for this tier. For a detailed test of whether the 12GB RTX 5070 can handle 34B at all, see can the RTX 5070 run 34B?
Check RTX 4090 Price→Buy on Shopee SG→70B parameter models (Llama 3 70B, Qwen 72B)
| Quantization | Download size | Minimum GPU |
|---|---|---|
| q8_0 | 75GB | 4x 24GB, or 2x 48GB |
| q6_K | 58GB | 2x 32GB, or 3x 24GB |
| q5_K_M | 50GB | 2x 32GB — two 4090s are 2GB short |
| q4_K_M | 43GB | 2x RTX 4090 (48GB) or A6000 |
| q3_K_M | 34GB | 2x 24GB. Does not fit a 32GB RTX 5090 |
| q3_K_S | 31GB | RTX 5090 (tight) |
| q2_K | 26GB | RTX 5090 32GB |
These are the published downloads for llama3:70b; Llama 3.3 70B ships at identical sizes. Budget KV cache on top of every row — a few thousand tokens of context is roughly 2-4GB at this model size, so a row that fits with 1GB spare does not really fit.
Verdict: 70B models are not practical on a single consumer GPU at good quality. Plan for dual GPUs, a workstation card, or cloud. For a specific look at what a 16GB mid-range card can do, see can the RTX 4060 Ti run Llama 70B? — and for the complete VRAM breakdown at every quantization level, see how much VRAM for a 70B model. For frontier MoE models like MiniMax M3, the resident weights dominate the VRAM math long before KV cache does — see our best GPU for MiniMax M3 breakdown.
Run 70B Models on RunPod Cloud→What is quantization?
Quantization reduces the precision of model weights to use less memory. The trade-off is quality:
| Level | Bits | Quality | Use Case |
|---|---|---|---|
| FP16 | 16-bit | Best | If VRAM allows |
| Q8 | 8-bit | Near-perfect | Default if you have room |
| Q6_K | 6-bit | Excellent | Best quality/size balance |
| Q5_K_M | 5-bit | Very good | Slight degradation |
| Q4_K_M | 4-bit | Good | Most popular for daily use |
| Q3_K_M | 3-bit | Acceptable | Noticeable on complex reasoning |
| Q2_K | 2-bit | Poor | Emergency only |
The sweet spot is Q4_K_M to Q6_K. Below Q4, quality degrades noticeably. Above Q6, the quality gains are minimal for most use cases. For a deeper dive into how quantization choices affect real-world output, see our best quantization for local LLM guide.
Context length eats VRAM too
VRAM usage isn’t just the model. The KV cache for context grows with conversation length:
| Context Length | Additional VRAM (7B) | Additional VRAM (13B) |
|---|---|---|
| 2,048 tokens | ~0.5GB | ~1GB |
| 4,096 tokens | ~1GB | ~2GB |
| 8,192 tokens | ~2GB | ~4GB |
| 16,384 tokens | ~4GB | ~8GB |
| 32,768 tokens | ~8GB | ~16GB |
This is why a model that “fits” in VRAM can crash during a long conversation. Always leave 2-4GB of headroom beyond the base model size.
GPU recommendations by budget
| VRAM Target | Best GPU | Price | Handles |
|---|---|---|---|
| 8GB | RTX 4060 | ~$479 | 7B quantized |
| 12GB | RTX 3060 12GB (used) | ~$250 | 7B full, 13B quantized |
| 16GB | RTX 4060 Ti 16GB | ~$425 | 7B-13B comfortably |
| 24GB | RTX 4090 | ~$2,200 | Up to 34B quantized |
| 32GB | RTX 5090 | ~$4,900+ | 34B comfortable, 70B only at q3_K_S or below |
| 48GB | 2x RTX 4090 or A6000 | ~$3,500 (A6000) / ~$4,400 (pair) | 70B at good quality |
Which GPU should YOU buy?
- Running 7B models only? Get an RTX 3060 12GB ($250 used) or RTX 4060 Ti 16GB ($425) for more headroom.
- Running 13B-34B models? Get an RTX 4090 ($2,200). Nothing else in consumer space gives you 34B at usable quantization on a single card.
- Future-proofing for 34B+ and experimental 70B? Get an RTX 5090 ($4,900). The only single consumer GPU that holds a 70B at all — and only at q3_K_S (31GB) or q2_K, where quality is visibly down. Treat single-card 70B as a demo, not a daily driver.
- Need 70B at good quality? Get 2x RTX 4090 ($4,400) or try cloud GPUs for occasional use.
Common mistakes
“I’ll just use CPU offloading” — Offloading layers to RAM works, but inference speed drops 5-10x for offloaded layers. A model that’s 50% on GPU and 50% offloaded is painfully slow. If your primary use case is local LLM summarization, long-document context is your binding constraint — plan VRAM for the KV cache first, not just the model weights.
“8GB is enough for everything” — It’s enough for 7B quantized. That’s it. If you plan to explore different models, get at least 12-16GB.
“I’ll buy the cheapest card with enough VRAM” — Memory bandwidth matters too. An old card with 16GB VRAM but low bandwidth will have much slower inference than a newer card with the same VRAM.
“More VRAM is always better” — True, but only up to what you need. Don’t buy a $3,500 A6000 if you only run 7B models. Wondering if a Mac Mini’s unified memory counts the same way? See our can the Mac Mini run LLMs guide for how Apple’s shared memory pool changes the VRAM math. For model-family specific breakdowns, see how much VRAM Qwen 3 needs if Qwen is your target, or how much VRAM Gemma 4 needs for Google’s latest.
Final verdict
| You want to run… | Get this GPU |
|---|---|
| 7B models (Mistral, Llama 8B) | RTX 4060 Ti 16GB ($425) |
| 13B models | RTX 4070 Ti Super 16GB ($800) |
| 34B models | RTX 4090 24GB ($2,200) |
| 70B models | 2x RTX 4090 or cloud GPU |
NVIDIA GeForce RTX 4060 Ti 16GB
16GB GDDR616GB VRAM at $425 is the new minimum for anyone serious about local LLM inference. Handles all 7B models and most 13B models comfortably.
Affiliate links — we may earn a commission at no extra cost to you. Amazon ships globally; Shopee SG covers Singapore & ASEAN.
NVIDIA GeForce RTX 4090
24GB GDDR6X24GB VRAM on a single card. The go-to choice for running 34B models at usable quantization levels.
Affiliate links — we may earn a commission at no extra cost to you. Amazon ships globally; Shopee SG covers Singapore & ASEAN.
16GB is the new minimum for anyone serious about local LLM inference. The landscape has shifted — 8GB cards are increasingly limiting as models grow.
Buy for the model size you want to run, not the model size you’re running today. VRAM is the one thing you can’t upgrade later.
Frequently Asked Questions
What LLMs can I run with 8GB VRAM?
With 8GB VRAM, you can run all 7B parameter models (Llama 3 8B, Mistral 7B, Gemma 7B) at Q4_K_M to Q6_K quantization. You can also fit some 7B models at Q8 with tight headroom. 13B models will not fit at any usable quantization level. Budget 2GB for context window overhead, so plan for models under 6GB in size.
What LLMs can I run with 16GB VRAM?
16GB VRAM handles all 7B models at full FP16 precision and all 13B models at Q4_K_M to Q8 quantization. Popular models like CodeLlama 13B, Qwen 14B, and Phi-3 Medium 14B all fit comfortably. You can also run 34B models at Q3_K_M in a pinch, though quality degrades noticeably at that quantization level. 16GB is the sweet spot for most local LLM users in 2026.
How much VRAM does Llama 4 need?
Llama 4 Scout (17B active parameters, 109B total in a mixture-of-experts architecture) needs roughly 67GB at Q4_K_M — the active parameter count governs speed, not memory, and all 109B parameters stay resident. That puts it past every consumer GPU; an 80GB A100 or a four-card 24GB build is the practical floor. Llama 4 Maverick (17B active, 400B total) needs roughly 245GB at Q4, which is cluster or cloud only.
Does quantization reduce VRAM usage?
Yes, dramatically. Quantization compresses model weights from 16-bit (FP16) down to 4-bit or lower, reducing VRAM usage by 2-4x. A 7B model uses about 14GB at FP16 but only 4.5GB at Q4_K_M. The quality trade-off is minimal between Q4_K_M and Q8 for most use cases — below Q4, quality degrades noticeably on complex reasoning tasks.