Home / Guides / Self-host vs API LLM cost: how to compare fairly

Self-host vs API LLM cost: how to compare fairly

A fair cost comparison between calling a hosted LLM API and running open weights on a rented or owned GPU, including utilization, ops, and quality caveats.

Why self-host vs API is two comparisons, not one

Teams usually ask whether self-host is cheaper when they need two answers: cost at volume, and whether open weights on your stack meet the quality bar the hosted API already clears.

Mixing them produces bad decisions. Open weights can be excellent for many workloads. They are still not a drop-in for frontier API quality. Price the cost race separately from evals. TokenCalculator handles the cost race with catalog API rates and editable GPU reference defaults.

What belongs on each side

API side: input tokens, output tokens, retries, and any cache or batch discounts your provider allows. Pull unit prices from the TokenCalculator catalog for the SKU you select. Do not invent live API unit prices from memory.

GPU side: rental $/hr × hours or purchase ÷ months, power, ops and eng time, serving stack work, and capacity headroom. GPU fields are editable reference defaults, not catalog-verified prices. Confirm with cloud vendors and your facility.

Fair comparison patterns

Same quality bar (hardest): you need frontier-class answers; self-host only wins if weights pass evals. Open weights vs open-model APIs: the cleanest cost race when you already accept that quality tier. Hybrid: API for hard prompts, self-host for high-volume easy traffic; break-even applies per lane.

Freeze the quality bar, measure movable monthly tokens, select the API SKU you would replace, enter editable GPU defaults, set realistic hours and tokens/sec, add ops overhead, then read API/mo, GPU/mo, delta, break-even, and capacity fit.

Hidden costs self-hosting adds

Idle nights on always-on rentals, multi-GPU complexity, model upgrades, safety filters you now own, storage and CI for serving images, compliance work, and opportunity cost of eng time.

These rarely show up in a naive $/hr comparison. Put a dollar or hour estimate in the ops field rather than pretending they are zero.

Common mistakes

Avoid these self-host vs API comparison mistakes.

  • Using frontier API quality as the goal while pricing only open-weight GPUs
  • Ignoring utilization and idle nights
  • Treating GPU presets as verified live prices
  • Forgetting output tokens on the API side
  • Skipping capacity and buying one GPU for a multi-GPU load
  • Jumping to purchase before a rental pilot

Frequently asked questions

Is it cheaper to self-host an LLM or use an API?
At low or spiky volume, APIs usually win. At high, steady volume with good utilization and acceptable open-weight quality, GPUs can win. Run your numbers in /tools/gpu-llm-cost.
Should I compare self-host cost to frontier or open-model APIs?
Match the quality bar. Frontier vs open weights is a product decision. Open-model API vs self-host is the cleaner cost decision.
What hidden costs does self-hosting add?
Ops time, idle hours, power, multi-GPU complexity, upgrades, observability, and compliance. Add them as overhead in the calculator.
At what volume does self-hosting beat API pricing?
Wherever monthly GPU cost undercuts monthly API cost with capacity still OK. Break-even moves with hours, tokens/sec, and GPU count.
Comparing Llama on GPU to Claude Opus is unfair, right?
For quality, yes if you assume parity. For cost education it can still be informative, but do not ship on cost alone.
Self-host looked cheap until we added eng time. How do I model that?
Put eng and on-call into the ops overhead field as dollars or hours × loaded cost. Re-read the delta.
Are GPU $/hr figures catalog-verified in TokenCalculator?
No. They are editable reference defaults. Verify with your cloud vendor. API rates use the catalog.
Our API bill is huge. Should we buy an H100?
First compare rental hours and utilization against the catalog API line. Buying is a second step after a rental pilot and a capacity plan.
Can cloud GPU rental beat both API and purchase?
Sometimes for medium horizons: less idle risk than ownership, potentially lower than high API volume. Model rental and purchase modes separately.
How does RAG change the comparison?
Large retrieval packs raise API input cost quickly. Estimate packs in /tools/rag-cost, then decide whether that generation volume belongs on API or GPU.

Try it in TokenCalculator

Set catalog API volume on one side and editable GPU rental or purchase defaults on the other, then read monthly delta and break-even.

Open GPU / LLM cost calculator · Open cost calculator · Estimate API spend · All guides

Related tools

GPU / LLM cost · API cost calculator · Cheapest API models

Related guides

GPU LLM cost calculator: compare API spend to self-host GPUs · GPU break-even · When to self-host · GPU utilization

Sources and references

Official documentation used for definitions, counting methods, or rate cards. Always confirm critical budgets on the provider page.