The AI Bill That Tripled in a Month, and Nobody Knew Why
By Ergini, Software & AI Developer
A composite story. The company and the people in it are invented. The problem, the rules and the system are real, and the full blueprint is in the use case library.
TL;DR
A composite story: an invented dispatch-software company in Toronto sees its OpenAI bill triple from July to August, and everyone blames the new AI agent. All four AI features share one API key, so nobody can check. The audit I would run tags every model call by feature and account, reconciles against the invoice and reads the most expensive requests in full. It finds the current time added to line 1 of a shared prompt, which stopped caching everywhere, one customer calling the agent in a loop, and staging on the production key. Every fix is checked against an eval set, and people own budgets and quality.
Which feature is this?
Farah's message in the leadership channel is two numbers and a question: "OpenAI, July: $7,480. August: $22,960. Which feature is this?"
Nobody answers for an hour, because nobody can. The company sells scheduling and dispatch software to about 900 home-service businesses, heating, plumbing and electrical contractors across Ontario and Quebec, from an office in Toronto. Over eighteen months it has shipped four AI features: job summaries that turn a technician's notes into a report and invoice lines, reply drafts in the shared customer inbox, an assistant that answers questions from equipment manuals and each customer's price book, and, since 4 August, a dispatch copilot, an agent that reads the day's schedule and proposes who goes where.
Everyone has the same theory. The copilot launched that week, it calls the model several times for every plan, and the daily chart in the OpenAI dashboard climbs from the same week. Owen, the CTO, has a feature flag ready to switch it off. Nadia, who owns the AI features, is against it: the copilot is the feature prospects ask about on sales calls.
Theo, who wrote most of the AI code, has spent three days in that dashboard. It shows spend by model and by day, and all four features share one API key in one project, so that is all it can show. Revenue grew a little in August. Farah has to tell the board whether the cheapest plan still makes money.
A step, not a curve
Owen, Farah and their company are invented; the order of work is the one I would follow on any bill like theirs. On the first video call I ask Theo to share that daily chart and zoom in on the first two weeks of August.
A feature that customers switch on one by one makes a curve, and there is a small one from the 4th, when the copilot launched. Then, on the 6th, a step: overnight, daily spend roughly triples, on the model all four features use, and it stays there. Something changed on the 6th, for everyone at once.
I ask for three things: the last three invoices with OpenAI's usage export, a list of every place in the code that calls a model, and read access to the repository. And I tell Owen what will not happen this week: nothing gets switched off, and no model gets swapped for a cheaper one. A cheaper setup nobody has measured does not fail loudly. Answers just get worse, and customers notice before anyone else does. Every step of the audit produces a number before anything changes.
Every call gets a name
The first change touches no prompt. Every model call in the product moves behind one internal function, which each feature calls with its own name and the customer's account. The function adds the plan, the prompt version and the model, and logs tokens in, cached tokens, tokens out, latency and retries to Langfuse and to the warehouse where the billing data already lives.
Nothing has been fixed yet, which is the point: whatever broke in August is still breaking, and now every call carries a name. After a week of traffic I reconcile the tagged total against OpenAI's own usage report for the same days. They should match, and they do not quite. A slice of spend has no tagged call behind it: the staging environment, whose nightly test suite runs every AI feature against the production key. It is not the tripling, but it is money nobody knew about, on a key that lives in too many places.
Then the breakdown, by feature, by account and by request. The copilot is about a fifth of the bill: real, but nowhere near a tripling, and most of it comes from one account, a franchise with forty locations that calls the copilot through the public API every few minutes to re-plan its day, eleven steps a run. Job summaries are the biggest line by far, and almost none of their input is billed as cached.
When Farah sees the bill split this way for the first time, her question changes from which feature to which customers.
Line 1 of the prompt
Token counts show that a prompt is big. Reading it shows why. I open the most expensive job summary of the week in Langfuse and read it top to bottom.
The system prompt is about 6,000 tokens, and most of it is twelve tool definitions the summary never calls. The template is shared with the reply drafts, the assistant and the copilot, which do use them. That has been true since spring, and it was cheap: the start of the prompt was identical on every call, and OpenAI bills a repeated prefix as cached input, at a fraction of the normal rate.
Line 1 is new. Since the deploy of 6 August, it holds the current time, to the second. Theo's commit message explains why: the copilot kept planning "tomorrow" on the wrong day, so he gave the shared template the time. Caching matches on the start of the prompt, so from that deploy on no two requests shared a prefix long enough to cache, and every feature paid full price for every input token, a dozen unused tool definitions included.
So everyone was right that the copilot caused the August bill, and wrong about how. The full list:
| What changed | What the tagged log shows | Who decides the fix |
|---|---|---|
| The time on line 1 of the shared template, from 6 August | Cached input falls to almost nothing on all four features at once | Code: the time moves to the end, and a test fails any template whose opening changes between calls |
| Twelve tool definitions in every job summary | Most of each summary's input is instructions and tools it never uses | Nadia, from an eval run of the trimmed prompt |
| The copilot, and one franchise calling it in a loop | About a fifth of the bill, most of it that one account | Farah and sales: a budget for the account |
| Staging on the production key | Invoice spend that no tagged call explains | Owen: one key per environment |
Three hundred job notes before anything changes
The cache fix is one line and carries almost no quality risk: the time moves to the end of the prompt, where the copilot still reads it. It ships first, with an alert on each feature's cache hit rate, because the next prompt edit could break it again just as quietly.
Everything else touches quality, so the eval set comes first: 300 real job notes, anonymized, each with the summary a dispatcher accepted, and a rubric Nadia writes. Does the summary keep the parts used, the fault found and the follow-up the customer was promised? Today's setup gets a baseline score. A judge model grades the runs, and its grades are first checked against Nadia's own on a sample. How to build a set like this is in the LLM evaluation guide.
Then each lever is priced on replayed traffic, cost and score side by side: the prompt without the unused tools, a cap on output length, and job summaries on a small model. The eval runs themselves go through the batch API, at half price with up to 24 hours to finish. The AI cost audit blueprint lists each lever with what it risks.
The small model is where testing earns its keep. Its average score is inside the tolerance Nadia set, but the misses are not spread evenly. Reading the worst cases, nearly all are French notes from customers in Quebec, where the small model drops part numbers and keeps the small talk. The average hid that. So French and mixed-language notes stay on the larger model by rule, a summary that fails its schema check is retried once on the larger model and logged, and Nadia reads a sample from each route every week. She approves each change from its eval run before it ships.
A pricing question, not a prompt
The franchise is not doing anything wrong. It found an endpoint that re-plans its day and calls it whenever a job moves, on a flat plan priced before the copilot existed. Code can enforce a budget per account. What that budget should be is for Farah and the account's sales rep to decide, and so is whether the franchise moves to a plan that includes heavy use.
The cap degrades instead of failing. Past its budget, the account's re-plans queue and run on a fixed interval, and the API says so in plain words instead of returning an error a developer has to guess at. The budgets live in code, where no prompt can argue with them, and the alerts are plain code too, comparing today with the last seven days for each feature.
The morning after the next release
A few weeks later, a release adds web search to the copilot. The next morning the cost channel in Slack has a message from the monitor: copilot cost per run is three times its seven-day median since the release, with the deploy hash and the team that owns it. Nobody waits for an invoice. Nadia checks whether the plans got better, Owen checks what it costs per account, and together they decide which plans get web search.
Farah gets a Monday digest: cost per active user for each feature, cost against revenue for the heaviest accounts, the eval score per route and how often small-model answers escalate. When the board asks whether the cheapest plan makes money, the answer comes from a query. And Theo spends his week on the product again.
Before you switch a feature off
Start with what you already have. Give each feature its own project or API key in the OpenAI or Anthropic console: it costs nothing, and next month's dashboard answers half of Farah's question. Helicone adds per-request cost with a change of base URL, and Langfuse adds traces, prompt versions and evals; the two are compared in LLM observability tools. For one or two AI features, that is enough, and I would set it up rather than build anything.
A layer of your own earns its place when cost has to be read against your own data, per account against what that account pays, and when budgets must slow a feature down instead of breaking it. The audit is a short, fixed-scope piece of AI integration work, usually in its first tier. The full blueprint has the order of work and the failure modes, and how providers price caching, batches and model tiers is in OpenAI API cost.
Frequently asked questions
Why did our OpenAI bill suddenly triple?
A sudden jump usually has a mechanical cause rather than growth: a prompt change that broke prompt caching, a new agent feature making several calls per action, one account calling a feature in a loop, a staging environment or cron job on the production key, or retries nobody counted. Tagging every call by feature and account, then reconciling against the provider's usage report, shows which one it is.
How do I find which feature is driving my AI costs?
Tag every model call where it leaves your code with the feature, account, plan, prompt version and model, and log tokens in, cached tokens, tokens out, latency and retries. Then reconcile the tagged total against the provider's usage report. Until that exists, giving each feature its own project or API key in the OpenAI or Anthropic console costs nothing and shows spend per feature.
Can a timestamp in the prompt break prompt caching?
Yes. Prompt caching matches on the start of the prompt, so one changed character near the top, such as the current time, a user name or tools in a different order, makes every request look new and pay full price. Keep instructions and tool definitions first and anything variable last, test templates for a stable prefix, and alert when a feature's cache hit rate drops.
Is it safe to move an AI feature to a cheaper model?
Only when it is measured. Build an eval set of a few hundred real, anonymized requests per feature with a baseline score, and let the product owner set the tolerance before anything changes. Route requests to the small model only where its score stays inside that tolerance, retry answers that fail validation on the larger model, and have a person read a sample from each route every week.