The question comes up in every scoping call, usually phrased as “what does this cost per month?” The honest answer is that token pricing is the part everyone models and the part that matters least. Here is what the bill actually looks like.
The four line items pilots forget
- Retries and tool loops. An agent that calls three tools and reconsiders once does not use one request's worth of tokens — it uses six to ten. Estimates built on a single round trip are routinely off by an order of magnitude.
- Retrieved context. Every retrieval stuffs thousands of tokens into the prompt. Retrieval quality is a cost lever, not just an accuracy lever: better ranking means fewer chunks, which means a smaller bill on every single call.
- Evaluation runs. A regression suite executed on every prompt change costs real money and is not optional. Skipping it is how a working system quietly degrades.
- Human review time. The queue is a feature, not a failure. Budget the hours, because a threshold set conservatively at launch sends a lot of work to a person.
Where self-hosting crosses over
GPU capacity is a fixed monthly cost; API usage is variable. That means self-hosting looks expensive at low volume and increasingly sensible as volume becomes steady and predictable. The crossover is a real calculation, not a philosophy, and it moves whenever either side changes its pricing.
Two things shift the maths before volume does. If a compliance requirement rules out external processing, the comparison ends before it starts. And if your workload is dominated by classification, extraction and routing rather than open-ended reasoning, a small open-weight model handles it at a fraction of the cost — you were paying frontier prices for work that did not need frontier capability.
The number that actually decides it
Not cost per run. Cost per correct outcome, compared against what the task costs today. A run that is wrong forty percent of the time is not cheap at any price, because someone downstream is checking all of it.
So the first question in a scoping call is never which model. It is: how many hours does this task consume now, what does an error cost, and what accuracy would make automation worth it? If the arithmetic does not work, we say so before anyone spends anything. Some of the most useful conversations we have end with a recommendation not to build.