Home / Guides / GPU utilization and LLM cost: why idle hours beat list $/hr

GPU utilization and LLM cost: why idle hours beat list $/hr

How GPU utilization, idle nights, and tokens/sec turn a cheap list $/hr into an expensive effective cost per token for self-hosted LLM inference.

Utilization is the missing denominator

List $/hr is a numerator. Effective cost per served token needs a denominator: useful tokens actually produced while you pay for the box.

If a GPU rents for a given $/hr but sits idle half the night, you still pay for those hours. Served tokens drop. Effective $/token rises. Break-even versus API moves further away. GPU fields in TokenCalculator are editable reference defaults; API monthly cost uses catalog rates.

Why utilization often matters more than list $/hr

API cost scales with tokens. GPU cost scales with uptime and ownership. A lower $/hr cannot save a schedule that bills large idle hours. Purchase amortization and ops overhead are mostly fixed until tokens flow.

Compare always-on vs business-hours with the same editable $/hr. The second can beat API sooner even if the hourly quote is identical.

Tokens/sec: the other half of utilization

Monthly tokens required ÷ productive seconds ≈ required average tokens/sec. Peaks need higher tokens/sec or queues. Undersized tokens/sec forces more GPUs, which multiplies $/hr and power.

A sane tokens/sec for 70B depends on GPU class, quantization, context length, batch size, and stack. Bench your shape. Until then, pick a conservative planning figure, not a promise.

Practical levers to raise utilization

Schedule rentals off when product allows. Fill valleys with batch jobs. Share a serving cluster across internal tools. Right-size models when quality allows. Avoid premature purchase until the utilization curve is real.

When utilization is unproven, rental usually risks less than buying. Self-hosting open weights is not a drop-in for frontier API quality, so do not raise utilization by forcing all traffic onto weak models.

Common mistakes

Avoid these utilization mistakes.

  • Using list $/hr as the only decision metric
  • Assuming 100% utilization in break-even slides
  • Confusing peak bench tokens/sec with monthly sustained throughput
  • Leaving rentals up just in case every night
  • Buying hardware before measuring idle fraction
  • Ignoring ops as a fixed cost against low utilization

Frequently asked questions

Does GPU utilization matter more than list $/hr?
Often yes. Idle hours and low tokens/sec dominate effective cost per token. Model hours explicitly in /tools/gpu-llm-cost.
How do I calculate GPU cost for LLM inference?
(Rental $/hr × hours) or (purchase ÷ months), plus power and ops. Divide by useful tokens for effective $/token.
Can one GPU handle my monthly token volume?
Only if sustained and peak tokens/sec cover demand. Check capacity OK/tight, not only monthly averages.
Idle GPU nights killed the savings story. What should we change?
Reduce billed hours, fill valleys with batch work, or stay on API until volume is steadier.
What’s a sane tokens/sec assumption for 70B?
Bench on your GPU, quantization, and context lengths. Use a conservative planning default until then.
At what volume does self-hosting beat API pricing?
At break-even for your hours and tokens/sec. Higher idle pushes that volume up.
Is renting a cloud GPU cheaper than buying?
When utilization is uncertain, rental usually risks less. Buy after utilization is proven.
Are GPU $/hr and power figures catalog-verified?
No. They are editable reference defaults. Verify with cloud vendors and your electricity rates. API rates use the TokenCalculator catalog.
Self-host looked cheap until we added eng time. Is that utilization?
It is fixed ops overhead. It hurts most when token utilization is low because overhead spreads across fewer tokens.
Our API bill is huge. Should we buy an H100?
Not before you simulate realistic hours and tokens/sec. A busy API bill with spiky traffic can still lose to idle GPUs. Prefer a rental pilot.

Try it in TokenCalculator

Set realistic hours and throughput against catalog API volume, then watch GPU/mo, break-even, and capacity shift.

Open GPU / LLM cost calculator · Open cost calculator · Estimate API spend · All guides

Related tools

GPU / LLM cost · API cost calculator · Cheapest API models

Related guides

GPU LLM cost calculator: compare API spend to self-host GPUs · Self-host vs API · GPU break-even · When to self-host

Sources and references

Official documentation used for definitions, counting methods, or rate cards. Always confirm critical budgets on the provider page.