You already know which model you want to run. The only question is which GPU to rent for it — so here is the whole answer up front.
| Model size | Rent this | VRAM | Approx $/hr (mid-2026) |
|---|---|---|---|
| 7B-13B (Q4) | RTX 4090 | 24GB | from ~$0.35 |
| 34B (Q4) | A100 80GB or RTX 6000-class | 48-80GB | ~$0.75-1.50 |
| 70B (Q4) | A100 80GB / H100 80GB, or 2x RTX 4090 | 80GB | ~$2.00-3.00 |
| 70B long context / 100B+ MoE | H200 | 141GB | ~$2.60+ |
| Frontier throughput | B200 | 192GB | ~$3.50 where available |
These are GPU-first cloud rates (RunPod, Vast.ai and similar marketplaces) as of mid-2026 — prices move weekly, so treat them as anchors, not quotes. Hyperscalers like AWS and GCP charge 2-3x more for the same silicon, and spot or interruptible pricing runs 50-80% below on-demand if you can tolerate getting reclaimed mid-run.
Check RunPod GPU pricing→Think in tokens per dollar, not dollars per hour
The hourly rate is the wrong number to optimize. What you actually buy is throughput, and a faster GPU that finishes sooner often costs less per million tokens than a cheap one that grinds all night.
A concrete example. An A100 80GB at roughly $1.50/hr pushing a 70B Q4 model through vLLM with batching can chew through a large summarization job in a third of the time a pair of budget cards would take. The meter runs at double the rate but for a third of the hours — you come out ahead, and you get your results the same afternoon.
The math flips for interactive chat. If you are the only user, a single conversation stream never saturates an H100, so most of that expensive compute sits idle between your keystrokes. Batch inference rewards big iron; single-user chat rewards the cheapest card that fits the model. Our tokens-per-second predictor lets you estimate throughput for a given GPU and model before you spend a cent.
7B-13B: rent an RTX 4090 and nothing bigger
A 13B model at Q4_K_M is a 7.9GB download. A 4090’s 24GB swallows that with room for a generous context window, and marketplace rates start around $0.35/hr — sometimes under $0.25/hr on Vast.ai when supply is loose.
I’ll say it plainly: most people renting H100s for 13B models are burning money. The H100 is maybe 2x faster for single-stream inference at 5-8x the price. Unless you are batch-serving hundreds of concurrent users, the 4090 wins tokens-per-dollar by a landslide at this size.
34B: the awkward middle tier
A 34B at Q4 technically squeezes into 24GB, but the context window gets claustrophobic fast. The comfortable move is a 48GB card — an RTX 6000 Ada-class rental at roughly $0.75-1.00/hr — or an A100 80GB at $1.10-1.50/hr if you want long context and headroom for KV cache.
My pick here depends on the job. Interactive coding assistant on a 34B? RTX 6000-class, cheapest that fits. Batch processing thousands of documents? A100, because the bandwidth advantage compounds across every request in the batch.
Browse Vast.ai marketplace rates→70B: A100 or H100, plus the 2x 4090 trick
A 70B model at Q4 wants roughly 40-45GB for weights plus KV cache, which means one 80GB card. On-demand A100 80GB rents for about $2/hr on GPU-first clouds; H100 80GB runs $2.50-3.00/hr and delivers meaningfully faster prefill and generation. If the job is throughput-bound, the H100 usually wins per-token despite the higher rate.
The budget route is two RTX 4090s with tensor parallelism, often $0.70-0.90/hr combined. It works well in vLLM and exllama-family stacks, though you eat some inter-GPU overhead and more setup friction. For the full hardware picture at this size — local and rented — see our best GPU for Llama 70B breakdown.
Frontier MoE and long context: H200 and B200
Once you cross into 100B+ MoE territory or need 128K+ context on a 70B, the 80GB cards run out of room for the KV cache. The H200’s 141GB is the practical answer at roughly $2.60+/hr — the extra capacity goes straight into context length and batch size.
The B200 at 192GB and around $3.50/hr is the throughput king when you can find one. Supply is still constrained mid-2026, so availability is spotty on marketplaces and queues are common. Rent it for serious batch workloads; don’t wait around for one just to run a chat session.
For provider choice at any of these tiers — reliability, pricing consistency, and which marketplace to trust with a long job — our RunPod vs Vast.ai comparison covers the tradeoffs in detail.
When renting is the wrong answer
Cloud GPUs shine for burst work: a weekend of experiments, a monthly fine-tune, occasional 70B runs. But if you run models every single day, the rental meter never stops and ownership starts winning fast. A daily 7B-34B workload on a rented 4090 at 4+ hours a day costs more per year than the card itself.
If that describes you, buy a 24GB card and rent only for the big stuff. The full break-even math is in our cloud vs local cost breakdown.
Check NVIDIA GeForce RTX 4090 on Amazon→Buy on Shopee SG→Fine-tuning is the exception where I rent even as a hardware owner. It demands far more VRAM than inference and runs in bounded bursts, which is exactly the workload rental pricing was made for — we cover the setup in renting a GPU for LLM fine-tuning.
Common mistakes that inflate your bill
- Renting an H100 for a small model. A 7B on an H100 is a sports car in a school zone. Match the tier to the model size in the table above and pocket the difference.
- Ignoring storage and egress fees. A 70B checkpoint is 40GB+ of downloads, and persistent volumes bill around the clock even while the GPU is off. Budget for data movement, not just compute.
- Leaving instances running idle. An H100 forgotten overnight is $25+ gone by morning. Set auto-stop timers, use serverless endpoints for spiky workloads, and check the dashboard before bed.
- Running spot instances without checkpointing. Spot pricing at 50-80% off is a genuine bargain right up until your instance gets reclaimed at hour nine of a ten-hour job. Save state every 30-60 minutes or pay for on-demand.
Final verdict
| Your model | Rent this | Expect to pay |
|---|---|---|
| 7B-13B | RTX 4090 24GB | from ~$0.35/hr |
| 34B | RTX 6000-class or A100 80GB | ~$0.75-1.50/hr |
| 70B | H100 80GB (throughput) or 2x 4090 (budget) | ~$0.70-3.00/hr |
| 70B long context / 100B+ MoE | H200 141GB | ~$2.60+/hr |
| Frontier batch throughput | B200 192GB | ~$3.50/hr |
Start on a marketplace, start one tier smaller than you think you need, and move up only when VRAM or throughput actually forces you to.
Spin up a cloud GPU on RunPod→Rent the smallest GPU that fits your model with context headroom — every dollar above that tier buys speed you should only pay for when batching.
Frequently asked questions
What is the best cloud GPU for LLM inference?
For most people, a rented RTX 4090 is the best value for 7B-13B models at roughly $0.30-0.45 per hour. Step up to an A100 80GB-class card for 34B, and an A100 or H100 80GB for 70B. The H200 only earns its rate for very long context windows or 100B+ MoE models.
How much does it cost to run a 70B model in the cloud?
Expect roughly $2-3 per hour for a single A100 or H100 80GB on GPU-first clouds like RunPod or Vast.ai, which runs a 70B model comfortably at Q4 quantization. A pair of RTX 4090s brings that down to roughly $0.70-0.90 per hour if you can handle tensor parallelism. Hyperscalers charge two to three times more for the same silicon.
Is a rented RTX 4090 good enough for LLMs?
Yes — for 7B-13B models it is the tokens-per-dollar king of the rental market, and it can squeeze a 34B at Q4 with a shorter context window. At roughly $0.30-0.45 per hour it beats every datacenter card on value for small and mid-size models. Rent bigger only when your model genuinely will not fit in 24GB.
Should I rent an H100 or H200?
Rent an H100 for 70B-class models at normal context lengths — it is cheaper and much easier to find. The H200 makes sense when you need its 141GB for long context windows, large KV caches, or 100B+ MoE models that overflow 80GB. If your workload fits in the H100, the H200’s premium buys you very little.