Home / Guides / When to self-host an LLM (and when to stay on API)
When to self-host an LLM (and when to stay on API)
A practical decision guide for moving inference to your own or rented GPUs versus staying on hosted LLM APIs, based on volume, utilization, quality, and ops readiness.
The decision in one paragraph
Self-host when three gates pass together: open weights (or licensed weights) meet your quality bar, monthly GPU cost undercuts the API line at realistic utilization, and your team can run serving without starving product work.
Fail any gate and staying on API (or switching to a cheaper hosted SKU) is usually wiser than buying GPUs. TokenCalculator quantifies the cost gate with catalog API rates and editable GPU reference defaults. Self-hosting open weights is not a drop-in for frontier API quality.
Gate 1: Quality and use case fit
Stay on API when you need frontier reasoning, multimodal features you do not run yourself, or rapid model upgrades without MLOps. Consider self-host when tasks are narrow and evals pass, data residency favors private inference, or you already run GPU workloads.
For cost-only races, compare against hosted open-model APIs from the catalog first before you assume self-host is the only lever.
Gate 2: Cost and capacity
Measure movable monthly tokens, select the API SKU you would replace, enter rental or purchase defaults from real quotes, set honest hours and utilization, enter defended tokens/sec, then read API/mo vs GPU/mo, break-even, and OK/tight.
Self-host is cost-plausible when volume is at or above break-even and capacity is OK (or funded for N GPUs). If RAG packs inflate API spend, fix retrieval shape before you conclude hardware is required.
Gate 3: Ops readiness and rent vs buy
Hidden costs include on-call, driver and serving upgrades, autoscaling experiments, safety filters, and capacity planning for spikes. If you cannot staff ops, do not self-host yet even if GPU math looks pretty.
Prefer cloud rental for testing break-even and spiky traffic. Prefer purchase only after rental proof and steady high utilization. Prefer API when volume is spiky, quality is frontier-only, or cheaper hosted SKUs and caching can fix the bill first.
Common mistakes
Avoid these self-host timing mistakes.
- Self-hosting for prestige instead of gates
- Ignoring idle utilization
- Treating GPU preset prices as verified
- Skipping evals while optimizing cost
- Buying hardware on day one
- Forgetting that open weights are not frontier APIs
Frequently asked questions
- When should I self-host an LLM?
- When quality, cost at real utilization, and ops readiness all pass. Use /tools/gpu-llm-cost for the cost gate.
- Is it cheaper to self-host an LLM or use an API?
- Sometimes at high steady volume. Often not at low or spiky volume. Run break-even rather than guessing.
- What hidden costs does self-hosting add?
- Eng time, idle hours, power, upgrades, observability, compliance, and multi-GPU complexity.
- At what volume does self-hosting beat API pricing?
- At your break-even point for a specific API SKU and GPU setup. There is no universal threshold.
- Should I compare self-host cost to frontier or open-model APIs?
- Match quality. Open-model API vs self-host is the fair cost race when open weights are acceptable.
- Is renting a cloud GPU cheaper than buying?
- Usually for pilots and uncertain demand. Buying can win later with proven utilization.
- Our API bill is huge. Should we buy an H100?
- Optimize API and run a rental break-even first. Buying is a late step.
- Comparing Llama on GPU to Claude Opus is unfair, right?
- For assuming equal product quality, yes. Cost can still be modeled, but ship only after evals.
- Are GPU prices in the calculator live vendor quotes?
- No. They are editable reference defaults. Verify with your cloud vendor. API side uses the TokenCalculator catalog.
- Idle GPU nights killed the savings story. Should we still self-host?
- Only if you can raise utilization (schedules, multi-tenant serving, batch jobs) or accept a higher break-even.
Try it in TokenCalculator
Compare catalog API monthly cost to editable GPU defaults, then use break-even and capacity fit as inputs to the go / no-go call.
Open GPU / LLM cost calculator · Open cost calculator · Estimate API spend · All guides
Related tools
Related guides
GPU LLM cost calculator: compare API spend to self-host GPUs · Self-host vs API · GPU break-even · GPU utilization
Sources and references
Official documentation used for definitions, counting methods, or rate cards. Always confirm critical budgets on the provider page.