Quick answer: You need at least 48GB of VRAM to run Llama 70B at usable quality — the q4_K_M build is a 43GB download. A single RTX 5090 (32GB) tops out at q3_K_S, where quality is visibly degraded, and no 24GB card holds a 70B at any published quantization (even q2_K is 26GB). For good quality you need dual GPUs or a workstation card like the A6000.
NVIDIA GeForce RTX 4090
24GB GDDR6XTwo RTX 4090s give you 48GB combined VRAM for Q4_K_M+ quality at ~$4,400. The best performance-per-dollar path to running Llama 70B locally.
Affiliate links — we may earn a commission at no extra cost to you. Amazon ships globally; Shopee SG covers Singapore & ASEAN.
The VRAM problem with 70B models
Llama 70B is one of the most capable open-source language models available, but it’s demanding. Here’s how much VRAM it actually needs:
VRAM capacity memory bandwidth Specs are manufacturer figures. Bar lengths are scaled independently per metric.
| Quantization | Download size | Quality Impact |
|---|---|---|
| fp16 (full) | 141GB | Best quality |
| q8_0 | 75GB | Near-lossless |
| q6_K | 58GB | Minimal loss |
| q5_K_M | 50GB | Slight loss |
| q4_K_M | 43GB | Noticeable on complex tasks |
| q3_K_M | 34GB | Significant degradation |
| q3_K_S | 31GB | Significant degradation |
| q2_K | 26GB | Major quality loss |
These are the published file sizes for llama3:70b — Llama 3.3 70B ships at the same sizes. Budget KV cache on top: a few thousand tokens of context costs roughly 2-4GB at this model size and grows with the conversation, so a quantization that fits with 1GB to spare does not really fit.
To see just how brutal the 70B math gets on consumer cards, try it live:
GPU options for Llama 70B
Single GPU options
| GPU | VRAM | Can Run 70B? | Best Quantization | Price |
|---|---|---|---|---|
| RTX 5090 | 32GB | Yes, limited | q3_K_S (degraded) | ~$4,900 |
| RTX 4090 | 24GB | No | none — q2_K is 26GB | ~$2,200 |
| A6000 | 48GB | Yes | Q4_K_M+ (good) | ~$3,500 |
| A100 80GB | 80GB | Yes | Q8+ (excellent) | ~$8,000+ |
Dual GPU options
| Setup | Total VRAM | Best Quantization | Approx Cost |
|---|---|---|---|
| 2x RTX 3090 | 48GB | Q4_K_M (good) | ~$1,640 used |
| 2x RTX 4090 | 48GB | Q4_K_M (good) | ~$4,400 |
| 2x RTX 5090 | 64GB | Q6_K (excellent) | ~$9,800 |
Best approaches by budget
Budget: Under $2,000 — Dual RTX 3090
The cheapest way to run Llama 70B at decent quality:
- 48GB combined VRAM handles Q4_K_M quantization
- RTX 3090s are widely available used for $800-900 each — see our dual RTX 3090 setup guide for the full build walkthrough
- Ollama and llama.cpp support multi-GPU splitting natively
- Inference speed is slower due to inter-GPU communication
Downsides: Needs a motherboard with two x16 PCIe slots, a beefy PSU (1200W+), and good case airflow. Two cards at 350W each generate serious heat.
Mid-range: $4,000-5,000 — RTX 5090 or dual 4090
Single RTX 5090: Simplest setup. Tops out at q3_K_S (31GB) — q3_K_M is 34GB and will not fit — so it runs, but you’ll notice quality loss on reasoning-heavy tasks. Best if you also use the GPU for smaller models where it excels. For tips on making the most of a single-card 70B setup, see how to run 70B on a single GPU, and for a broader look at the $4,900 tier our best GPU for LLM under $2,000 guide ranks the alternatives.
Dual RTX 4090: 48GB total VRAM for Q4_K_M+ quality. Better output quality than a single 5090, but more complex setup and higher power draw.
High-end: $3,500+ — NVIDIA A6000
The NVIDIA A6000 with 48GB VRAM on a single card is the cleanest solution:
- Runs q4_K_M (43GB) on one card with room for context; q5_K_M at 50GB does not fit
- No multi-GPU complexity
- Professional-grade reliability
- ECC memory for consistent results
The downside is price and availability. The A6000 is a professional card with professional pricing.
Ollama setup for multi-GPU
If you go the dual-GPU route, Ollama handles GPU splitting automatically:
OLLAMA_NUM_GPU=999 ollama run llama3:70b-q4_K_MFor llama.cpp, specify the split:
--tensor-split 24,24Both tools will distribute model layers across available GPUs. Inference speed scales roughly 60-70% of linear with two cards due to communication overhead.
Inference speed expectations
| Setup | Llama 70B Q4_K_M | Tokens/sec |
|---|---|---|
| Single A6000 (48GB) | Full model on GPU | ~15-20 tok/s |
| 2x RTX 4090 (48GB) | Split across GPUs | ~12-18 tok/s |
| 2x RTX 3090 (48GB) | Split across GPUs | ~8-12 tok/s |
| Single RTX 5090 (Q3) | Degraded quality | ~18-22 tok/s |
| CPU offload (partial) | Slow | ~2-5 tok/s |
These are approximate for 2048 context length. Longer contexts reduce speed.
Should you even run 70B locally?
Before investing in hardware, consider:
- Is 70B actually better for your use case? For many tasks, a well-prompted 13B or fine-tuned 34B model performs nearly as well.
- Would cloud be cheaper? If you only need 70B occasionally, cloud GPU rental (RunPod, Vast.ai) at $1-2/hour may be more cost-effective than a $3,000+ hardware investment. See RunPod vs Vast.ai for LLM to understand which platform offers better pricing and reliability for this workload, and our cloud GPU TCO vs self-hosted LLM breakdown for the exact monthly break-even math.
- Do you need the privacy? Local inference means your data never leaves your machine. If that matters, the hardware cost is justified.
Which GPU should YOU buy for Llama 70B?
- Running 70B as your primary model? Get 2x RTX 4090 ($4,400). 48GB combined VRAM handles Q4_K_M with good quality and decent speed.
- Running 70B occasionally alongside smaller models? Get an RTX 5090 ($4,900). Handles 70B at q3_K_S and excels at 7B-34B models the rest of the time.
- Need the best single-card 70B experience? Get an NVIDIA A6000 ($3,500). 48GB on one card means Q4_K_M+ without multi-GPU complexity.
- Only need 70B sometimes? Use cloud GPUs instead. $1-2/hour beats a $3,000+ hardware investment for occasional use.
Common mistakes to avoid
- Buying a single 24GB GPU expecting to run 70B — it does not work at any published quantization. The smallest Ollama ships, q2_K, is 26GB against the 4090’s 24GB. You need 32GB minimum, and realistically 48GB for good results.
- Ignoring memory bandwidth in dual-GPU setups — inter-GPU communication adds latency. Two RTX 3090s (936 GB/s each) outperform two RTX 4060 Tis even if total VRAM is similar, because bandwidth determines token generation speed.
- Not accounting for context length VRAM overhead — at q4_K_M, Llama 70B is 43GB of weights alone. A 4K context window adds about 1.25GB of KV cache — 70B models use grouped-query attention, so the cache is far smaller than the weights suggest. Plan your VRAM budget accordingly. For a full breakdown of exactly how much VRAM each 70B quantization level needs, see how much VRAM for a 70B model.
- Skipping the “do I actually need 70B” question — a well-quantized 34B model fits a single RTX 4090, which no 70B does, and often matches a heavily-quantized 70B in output quality, at 3x the inference speed and half the hardware cost. Llama 4 Scout beats Llama 3 70B on benchmarks but is not a lighter option — at ~67GB for Q4_K_M it needs more memory than 70B does; see our Llama 4 Scout GPU guide for what it actually takes. DeepSeek’s reasoning-tuned 32B is another single-card alternative — see our DeepSeek GPU guide for VRAM needs and tok/s on 24GB cards. If you are wondering whether a budget card like the 4060 Ti can even attempt 70B, see can the RTX 4060 Ti run Llama 70B?
Final verdict
| Situation | Recommendation |
|---|---|
| Must be single GPU | NVIDIA A6000 (48GB) |
| Best value | 2x RTX 3090 used (~$1,640) |
| Best performance/value | 2x RTX 4090 (~$4,400) |
| Occasional 70B use | Cloud GPU (RunPod/Vast.ai) |
| Mostly smaller models | RTX 5090 single card |
NVIDIA GeForce RTX 4090
24GB GDDR6XTwo of these give you 48GB combined VRAM for Q4_K_M+ quality. The most popular dual-GPU setup for running Llama 70B locally.
Affiliate links — we may earn a commission at no extra cost to you. Amazon ships globally; Shopee SG covers Singapore & ASEAN.
For most people, Llama 70B is not a single-GPU workload at consumer prices. Accept that and plan for either dual GPUs, a workstation card, or cloud.
The best GPU for Llama 70B is the one that gives you enough VRAM to avoid aggressive quantization. Quality degrades fast below Q4 — don’t sacrifice output quality to save on hardware.
Llama 70B FAQ
Can four RTX 3060 12GB cards run Llama 70B at Q4?
Four 12GB cards give you 48GB combined, which covers the 43GB that q4_K_M actually downloads as. But memory bandwidth determines token generation speed, and inter-GPU communication adds latency with every extra card — the same reason two RTX 3090s outperform stacks of slower cards with similar total VRAM. Expect speeds noticeably below the roughly 8-12 tok/s of a dual RTX 3090 setup, which remains the better budget path.
What are the GPU requirements for Llama 2 70B?
The 70B-class Llama models are a 43GB download at q4_K_M, and KV cache sits on top of that, so plan for 48GB for good quality — dual RTX 3090s, dual RTX 4090s, or a single A6000. A 24GB card like the RTX 4090 only fits Q2_K, where output quality degrades significantly, and a 32GB RTX 5090 manages Q3_K_M with noticeable quality loss on reasoning tasks.
How many tokens per second does Llama 70B run at locally?
It depends heavily on the setup. A single A6000 with the full Q4_K_M model on one card manages roughly 15-20 tok/s, dual RTX 4090s land around 12-18 tok/s, and dual RTX 3090s around 8-12 tok/s. A single RTX 5090 at Q3 runs faster, roughly 18-22 tok/s, but with degraded quality. Partial CPU offloading collapses to roughly 2-5 tok/s.
Is it cheaper to run Llama 70B in the cloud?
Often, yes — if your usage is occasional. Cloud GPU rental on RunPod or Vast.ai runs roughly $1-2/hour, which beats a $3,000+ dual-GPU investment unless 70B is your daily driver. Local hardware wins on privacy, since your data never leaves your machine, and on long-term cost once you’re running the model heavily every day.