AI Systems
Self-hosting your LLM probably costs more
Self-hosting an open model is supposed to be the cheap option. Run the numbers and it usually isn't - and the reason is one variable almost nobody measures.
The open-weights argument is usually made on price: the model is free, so running it yourself must be cheaper than paying per token. The weights are free. The GPU is not, and the GPU is what you are actually buying.
Once you write the cost per million tokens as a single expression, the whole argument reduces to one number — and it is not a number about models at all.
The expression everything reduces to
cost_per_million = (dollars_per_hour)
/ (tokens_per_second * 3600)
* 1_000_000
dollars_per_hour is what the hardware costs whether or not you use it. tokens_per_second is what you actually get out of it. Every argument about self-hosting economics is an argument about one of those two terms, and most of them quietly assume the best case for both.
Why the throughput number is usually wrong
This is the term people get wrong by two orders of magnitude, because the number quoted in benchmarks and the number you get in production are measuring different things.
| tokens/second | |
|---|---|
| batched, many concurrent requests | ~3,066 |
| one user, one stream | ~24 |
Both are real numbers for an H100 running a 70-billion-parameter model. The difference is batching. Generating tokens for one stream leaves the card overwhelmingly idle — it is waiting on memory bandwidth, not compute — so serving many requests at once is nearly free compared to serving one.
Which means the benchmark figure is achievable only if you have enough simultaneous traffic to fill the batch. If you do not, you are paying for a card running at a fraction of a percent of its capacity, and your cost per token is the idle cost divided by the handful of tokens you actually produced.
The number that decides it
So the question is not "is the model good enough". It is:
| you would have paid | utilisation needed to break even |
|---|---|
| the cheapest mainstream API | ~34% |
| a mid-tier API | ~4% |
| a frontier API | ~1.6% |
Read that carefully, because it is the opposite of what the framing usually implies. Against the cheapest API you need to keep a card meaningfully busy about a third of the time, every hour, indefinitely. Against expensive frontier pricing the bar is low — under two percent — which is why the self-hosting argument is strongest precisely where people make it least.
"Sustained" is doing real work in that sentence. It is not your peak hour. It is the average over the whole billing period, including nights, weekends, and every week nobody used the feature.
The costs that do not appear in the comparison
| monthly | |
|---|---|
| H100, running 24/7 | ~$1,901 |
| 25% of one engineer | ~$4,000 |
The engineer is the larger number, and it is the one that never appears in the blog post comparing token prices. Somebody has to own model updates, driver and CUDA versions, inference server upgrades, capacity planning, and being paged when it stops.
Add it in and the all-in comparison at 100% utilisation — the most generous possible assumption — looks like this:
| $ per million tokens | |
|---|---|
| self-hosted, all-in | ~0.743 |
| cheapest API | ~0.700 |
At a hundred percent utilisation, which nobody achieves, it is still slightly more expensive than simply buying the tokens.
Translated into a realistic monthly volume:
| monthly | |
|---|---|
| self-hosted, all-in | ~$5,901 |
| cheapest API | ~$105 |
Where the argument actually holds
None of this says self-hosting is wrong. It says the reason people give for it is usually wrong.
Self-hosting wins on data residency, on latency floors you control, on not being deprecated by a vendor, and on sustained high-volume workloads that genuinely keep the batch full. Those are good reasons and they survive scrutiny.
"It is cheaper" is not one of them, unless you can name your sustained utilisation figure — and if you can, you already know which side of the line you are on.
Spot instances at around $2.10/hour against on-demand at around $2.64 shift the break-even point but do not move it to a different kind of place. You are still buying hours and selling tokens, and the ratio between them is still set by how full you can keep the card.
What to look at next
Before comparing any two options, compute your own utilisation: total tokens generated last month, divided by what one card could have produced running flat out for those hours. Most teams have never calculated it, and the number is usually far lower than the intuition.
Next: local inference does not remove the re-read multiplier either — we worked through where the bill actually comes from in episode three.
