Use caseAI product featureReporting and analytics
An AI bill growing faster than your users: a cost audit, model routing and budgets
For SaaS teams whose OpenAI or Anthropic bill outgrew usage: cost per feature and customer, caching, batching, routing, and evals that keep quality intact.
A blueprint, not a client story. The business described is illustrative; the architecture, integrations and trade-offs are real, and this is how I would build it. By Ergini, .
The short version
An audit and fix for SaaS products whose model bill grows faster than usage. Every OpenAI and Anthropic call is tagged by feature, customer and prompt version and logged to your warehouse, so cost can be read per feature and per account. Then the levers: working prompt caching, batch APIs for offline jobs, trimmed prompts and retrieval, output caps, and routing simple requests to smaller models, each checked against an eval set. Budgets alert in Slack; people set the quality bar and approve every change.
- Best for
- SaaS teams whose model bill grows faster than usage and who cannot say which feature or customer is driving it.
- Connects to
- OpenAI and Anthropic APIs, Langfuse or Helicone, AI gateway, Data warehouse, Slack
- The AI does
- A small model routes simple requests away from the expensive one, and a judge model grades eval runs so every cost change is checked against quality.
- People do
- The product owner sets the quality bar per feature, approves routing and prompt changes from the eval results, and decides each customer's budget.
- Built as
- AI Integration, usually $3.5K - $15K
Usage up 40%, the model bill up 200%
Take a B2B SaaS for property managers with about 1,500 customer accounts. Over eighteen months it shipped four AI features: an assistant that answers questions over each customer's documents, a summary of every tenant ticket, a weekly portfolio report, and, last quarter, an agent that drafts replies and looks things up. Weekly active users grew 40% in nine months. The monthly model bill went from about $6,000 to $18,000.
Finance asks the obvious questions: which feature, which customers, and does the cheapest plan still make money? Nobody can answer. All four features share one API key, the provider dashboards show spend by model and by day, and the engineer who shipped the features now spends part of every week on the bill. Managing the AI has quietly become a job of its own.
This is the normal shape of 2026. Epoch AI estimates that the price of a fixed level of model capability falls between 9 and 900 times a year, depending on the task, yet bills rise because products spend more tokens on every action. Visibility lags behind: in KPMG's AI Quarterly Pulse for Q2 2026, only 26% of large US companies reported full, real-time visibility into what their AI systems cost to operate.
The bill is rarely one mistake. It is prompts that grew an instruction at a time, a cache that stopped hitting, a nightly job paying daytime prices, and a flagship model doing work a small one would do. Each is fixable. The hard part is fixing them without the product getting quietly worse.
The audit, in the order I run it
Every step produces a number before anything is changed.
- Tag every model call where it leaves your code (feature, account, plan, prompt version, model) and log tokens in, cached tokens, tokens out, latency and retries.
- Reconcile the tagged total against the providers' own usage and cost reports. The gap is usually a forgotten cron job, a staging environment on the production key, or retries nobody counted.
- Break cost down per feature, per account and per request, and find the tails: the few accounts, prompts or agent loops that carry most of the bill.
- Read the most expensive requests in full. Token counts show that a prompt is big; reading it shows why: duplicated instructions, tool definitions a summary never uses, a whole conversation resent every turn.
- Build the eval set before changing anything: a few hundred real, anonymized requests per feature with reference answers or rubrics, and a baseline score for today's setup.
- Price each lever on replayed traffic, with cost and eval score measured side by side.
- Rank the changes by saving and by quality risk, and agree the order with the product owner.
The result is a ranked list with savings and quality impact measured on your own traffic, not a rule of thumb.
Inside the audit report, and the alert that followed
Queries against the call log for the property-management SaaS above, then one of the alerts the build leaves behind.
Warehouse queries and a Slack alert, illustrative SaaS
- cost_by_feature(month: "2026-08")assistant 41% / ticket summaries 27% / reply agent 19% / weekly report 10% / untagged 3%
- tokens_per_call(feature: "ticket_summary", since: "2026-03")median input 7,900 tokens, was 2,100 / 5,600 of them are the system prompt and 14 tool definitions the summary never calls
- cache_hit_rate(feature: "assistant")0.4% of input tokens cached / line 1 of the system prompt holds the current timestamp
- top_accounts(feature: "reply_agent")3 accounts carry 38% of agent cost / one calls it through the API in a loop, 11 steps per run
- eval_run(feature: "ticket_summary", config: "small model, trimmed prompt")score 4.31 against baseline 4.36, inside the 0.10 tolerance / 2 of 300 cases worse, both German tickets with attachments
- Summaries move to the small model, except tickets with attachments.
Cost monitor · Slack, #ai-costs
reply_agent: cost per run is 3.1x its 7-day median since yesterday 14:00, matching deploy 8f2c1 (adds web search). Today's budget is 71% used at 11:00. Owner: platform team.
The levers, and what each one risks
Pulled roughly in this order, because the first carry almost no quality risk and the last carry the most.
| Lever | What changes | What can go wrong | How it is checked |
|---|---|---|---|
| Fix the cache prefix | Static instructions and tools first, anything variable last, so cached input bills at the reduced rate | A later prompt edit silently breaks it again | Cache hit rate per feature, with an alert on a sudden drop |
| Cap and shape output | Token limits, and structured outputs wherever code reads the result | Truncated answers on long cases | Truncation rate and eval score on the longest cases |
| Batch the offline work | Reports, backfills and eval runs move to the batch APIs at half price, with up to 24 hours to finish | A batch misses its window | Completion time per batch, with a synchronous fallback |
| Trim prompts and retrieval | Unused tools and repeated instructions removed; fewer, better chunks with a reranker | An answer loses context it needed | Eval score, plus citation checks on the assistant |
| Route to smaller models | Simple requests go to a small model; hard or failed ones escalate | Hard cases judged easy | Eval score per route and a weekly sample a person reads |
| Budgets and caps | Limits per feature and per account, degrading gracefully | A good customer hits a cap mid-task | Cap events per account, reviewed with the account owner |
What happens to one model call after the audit
Most of this path is ordinary code. The model's part is small: a cheap classifier that picks a route, and the model that does the work.
01 Trigger · Internal AI client
A feature asks for a completion
The summary job, the assistant or the agent calls one internal function instead of a provider SDK, passing the feature name and the account.
02 Plain code · Gateway, budget table
Tag it and check the budget
The call gets feature, account, plan and prompt-version tags. Over budget, it degrades by rule: a smaller model, a queued job, or a clear limit message.
03 AI model · Small model, structured output
Pick a route
Rules route the obvious cases. For the rest, a small model labels the request simple or hard under a strict schema, in a handful of tokens.
04 Decision
Which model gets it?
The routing table is versioned code, and every change to it is eval-checked.
- Simple, and the feature's eval allows the small model then small model
- Hard, long context, or a plan that includes the larger model then larger model
- Provider error or timeout then fallback provider, same prompt, with a retry limit
05 System · OpenAI or Anthropic API
Call the provider with a stable prefix
Instructions and tool definitions come first and never contain timestamps or names, so the prefix cache keeps hitting. Output has a token cap.
06 Plain code
Validate, and escalate once
Structured output is checked against its schema. A small-model answer that fails is retried once on the larger model, and the escalation is logged.
07 Result · Langfuse or Helicone, warehouse
Log the cost
Tokens, cached tokens, cost, latency, route and outcome go to the trace store and the warehouse, where the nightly report and the anomaly checks run.
08 Person · Slack
Owners review the week
Each feature owner gets a digest: cost per active user, eval score per route, escalation rate and a sample of routed requests to read. Route and budget changes need their approval.
Where cost cutting goes wrong
Each of these saves money on the invoice and spends it somewhere else.
A cheaper setup that is quietly worse
A smaller model or a shorter prompt passes a spot check and fails on German tickets, long threads or cases nobody tried. Nothing errors; answers just get worse, and customers notice first. Every change runs against the eval set with a tolerance agreed in advance, and the weekly sample catches what the set misses.
A cache that silently stops hitting
Prompt caching matches on the prefix: one changed character near the top, such as a date, a user name or tools in a different order, and every request pays full price again. Prompt templates are tested for a stable prefix, and a drop in hit rate alerts the owning team the same day.
Stale answers from a response cache
Caching whole responses is riskier than prompt caching: an answer about a tenant's balance cached yesterday is wrong today. Response caches are keyed on the prompt version and the data they read, expire with that data, and never serve per-account answers unless the key includes the account.
Routing mistakes that look like success
A router that sends nearly everything to the small model looks wonderful in the cost report. What matters is quality per route: an eval score for each, the rate of escalations after failed validation, and requests a person reads every week.
Savings that move the cost elsewhere
Fewer retrieved chunks cut tokens per answer, and can raise the number of follow-up questions when answers lose context. Cost is measured per resolved task, not per call, so a change that makes users ask twice shows up as the loss it is.
Lock-in, to a provider or to the gateway
Prompts are tuned to a model, so switching providers is an eval run, not a config change; keeping tags, evals and routing in your own repository makes that run possible. A gateway holds every provider key, which makes it a target: the open-source LiteLLM package was backdoored on PyPI in March 2026. Versions get pinned and keys scoped.
Models, code and owners: who holds which decision
The AI model
Label a request simple or hard
A small model does it in a few tokens, and failed validation catches its misses.
Grade eval outputs against a rubric
It scales grading; its scores are first checked against a person's grades on a sample.
Plain code
Tag every call and reconcile with the invoice
Attribution is only trusted when it is complete, and only code is complete.
Enforce budgets and caps
A limit that a prompt can argue with is not a limit.
Escalate on failed validation
Schema checks are cheap and exact, so they decide when a small model's answer is not good enough.
Spot anomalies and post alerts
Today against the last seven days, per feature. Nothing clever, nothing missed.
A person
Set the quality tolerance per feature
What counts as good enough is a product decision, not an engineering one.
Approve routing and prompt changes
From the eval diff and the sample, before the change ships.
Set and raise account budgets
A heavy account is a pricing conversation, which belongs to sales and finance.
Is a dashboard enough, or do you need your own layer?
Start with what you already have. The OpenAI and Anthropic consoles show spend by model, day, project and API key, and giving each feature its own project or key costs nothing and answers half of finance's first question. Helicone adds per-request cost with a base URL change, Langfuse adds traces, prompt versions and evals (compared in LLM observability tools), and OpenRouter or a managed gateway puts several providers behind one API with fallbacks. For one or two AI features, that stack is enough, and I would set it up rather than build anything.
A custom layer earns its place when cost has to be read against your own data: cost per account against what that account pays, per plan, per workspace. It also wins when routing depends on context no gateway can see, such as the customer's plan, the document type or whether the request came through your API, and when budgets must degrade features gracefully instead of failing requests. No off-the-shelf tool knows your pricing page.
Most builds are hybrid: a thin layer in your code that tags, routes and enforces budgets, with Langfuse or Helicone underneath for traces and a gateway for failover. The tags, the eval set and the routing table stay in your repository and warehouse, which keeps a future change of provider an eval run rather than a rewrite. The same layer is the start of a private AI gateway if your staff use models too.
How you would know it is working
A blueprint has no results to report, so here is what I would measure from the first week instead, on your own data.
- Cost per active user, per feature
- Monthly spend on each feature divided by its active users, which separates growth from waste.
- Cache hit rate
- The share of input tokens billed as cached, per feature. A sudden drop almost always means a prompt change broke the prefix.
- Eval score per route
- Each feature's score on the fixed eval set, per model route, rerun on every prompt, model or routing change.
- Unattributed spend
- The gap between the provider invoice and the sum of tagged calls. It should shrink to almost nothing and stay there.
- AI cost against revenue, per account
- For the heaviest accounts, so plan limits and pricing are decided on data.
What a build like this costs
This is built as AI Integration, which runs $3.5K - $45K overall. A build like this one usually lands in the single AI feature tier: $3.5K - $15K, 1-3 weeks. The first working version runs on your real data well before the end of that window.
What it costs to run
Small next to the model bill. Self-hosted Langfuse costs its hosting; Helicone and Langfuse Cloud have free tiers and volume pricing; a gateway is a managed plan or a small service. Eval runs are model calls too: a few hundred graded cases per change is cheap, but nightly full runs on every feature add up, so they go through the batch APIs.
What moves the price
- How many AI features, providers and API keys have to be traced and attributed
- Whether per-account attribution needs joins to tenant, plan and billing data in the warehouse
- The effort of building eval sets: some features have clear right answers, others need rubrics and graded examples
- Whether routing and budgets live in a gateway, in your own code, or both
Who this is for
- SaaS companies whose model bill has grown faster than active users for two quarters running
- Teams running several AI features on one shared API key, with no view per feature or per customer
- Products on flat-rate plans where a few heavy accounts may be eating the margin on everyone else
- Finance and product leads who need a forecast of AI cost before repricing a plan
Questions people ask about this
How do I reduce my OpenAI API costs?
Measure first, per feature and per account, because the fix depends on where the money goes. Then pull the low-risk levers: a stable prompt prefix so caching works, output caps, and the Batch API for anything that can wait. Trimming prompts and retrieval and routing simple requests to smaller models come next, each only after an eval set shows quality holds.
Why is our LLM bill growing faster than our users?
Usually because tokens per action grew, not the number of actions. System prompts gain an instruction at a time, tool lists grow, conversation history is resent in full, retrieval passes more chunks, and agent features make several calls per click. Retries multiply all of it. Falling token prices hide the growth for a while, until a new feature or a heavy customer tips the bill.
What is model routing, and is it safe?
Model routing sends each request to the cheapest model that can handle it: usually a small model for simple, well-defined work and a larger one for the rest. It is safe when the routes are measured, with an eval score per route, validation that escalates a failed small-model answer, and a weekly sample a person reads. Without those, routing saves money by making the product worse.
Does prompt caching work with both OpenAI and Anthropic?
Yes, when the start of the prompt stays identical from call to call. OpenAI caches long prompts automatically, and Anthropic lets you mark exactly which part to cache; both bill cached input at a fraction of the normal rate. The usual reason caching does nothing is a timestamp, a user name or a reordered tool list near the top of the prompt, which makes every request look new.
How much does an AI cost audit cost, and when does it pay?
The audit is a short, fixed-scope piece of AI integration work, shown in the price block on this page. Whether it pays depends on your bill: at a few hundred dollars a month, a dashboard and an afternoon are enough, and I will say so. It pays when AI is a real line in your margin, because every change is priced on your own traffic before you commit to it.
Sources