01
the bill follows the design
Teams that run agents or multi-turn LLM features tend to discover the same thing a few months in: spend grows faster than usage, and the usual responses (per-seat caps, monthly ceilings, a dashboard that turns red) treat the symptom. Gartner's June 2026 forecast that AI coding costs will surpass the average developer's salary by 2028 is about coding assistants, where you mostly control usage policy. In a product feature you also control the architecture, and that's where most of the money is decided: how much context each call carries, how often the same tokens are paid for again, and which model answers.
This post works through those decisions with numbers. The prices are from Anthropic's and OpenAI's pricing pages as of 8 November 2026, and the workload shapes are assumptions, so treat the scripts as templates and plug in your own token counts from logs.
02
where the tokens go in an agent loop
An agent built on a stateless chat API resends the whole conversation on every turn. The system prompt and tool definitions ride along each time, and every tool result the agent has seen stays in the history for all later turns, so the input billed per task grows roughly with the square of the number of turns. That's the mechanism behind most bills that look absurd next to the traffic they serve.
Here's a model of one task: a 13,000-token prefix (system prompt, tool definitions and the user's request), 20 turns, 300 output tokens per turn and a 2,000-token tool result after each one, priced at Claude Sonnet 5.5's rates ($2 per million input tokens, $2.50 for 5-minute cache writes, $0.10 for cache reads, $10 for output). The cached variant assumes each turn reads everything up to the previous turn from cache and writes only what's new.
Without caching, the task bills 697,000 input tokens to produce 6,000 output tokens, and input is 96% of the cost. Caching cuts it by about 80% because reads cost 5% of the base input price on this model. What's left is dominated by cache writes, which is why the third line matters: once caching is on, the size of what each turn adds becomes the lever, and a tool that returns 400 relevant tokens instead of 2,000 raw ones takes off another 39%.
03
prompt caching, priced correctly
Both big providers cache prompt prefixes, and the pricing rules decide how you should lay out a request. On Anthropic, a 5-minute cache write costs 1.25 times the base input price, a 1-hour write costs 2 times, and a read costs 0.1 times (0.05 times on Claude Opus 5.5 and Sonnet 5.5). These multipliers stack with the batch discount. On OpenAI, caching is on by default for supported models, the minimum cacheable prompt is 1,024 tokens on GPT-5.6 and later, reads cost 0.1 times the input rate (0.05 times on GPT-6.1 Sol and Terra), and on GPT-5.6 and later a write costs 1.25 times for the default 30-minute lifetime or 2.5 times for 12 hours.
Caching only works on an identical prefix. Anthropic builds the cache key from tools, then system, then messages, in that order, and a change at one level invalidates that level and everything after it, so editing a tool description throws away the whole cache. You can mark up to four breakpoints with cache_control, or put a single cache_control at the top level and let the API move the breakpoint to the last cacheable block on every request, which is what you want in a growing conversation:
The explicit breakpoint on the system block keeps the shared prefix cached across different conversations, and the top-level one follows each conversation as it grows. Watch the 20-block lookback rule: the API looks back at most 20 blocks from a breakpoint for an earlier cache entry, so a turn that appends more than 20 blocks needs an extra breakpoint. Verify all of this in production through the usage fields, cache_creation_input_tokens and cache_read_input_tokens on Anthropic and usage.input_tokens_details.cached_tokens on OpenAI, because a prefix that silently changes on every request (a timestamp in the system prompt, tools serialized in a different order, a user name interpolated near the top) costs you the write premium without ever producing a read.
04
context is a quality budget too
Cutting context saves money and usually improves answers. Chroma's "Context Rot" report from July 2025 tested 18 models and found that performance degrades as input grows, even on simple tasks and well inside the context window. On a LongMemEval-based test, models did clearly better with a focused prompt of about 300 tokens than with the full 113,000-token version containing the same answer, and topically similar distractor passages hurt accuracy further. That fits the older "Lost in the Middle" result (Liu et al., TACL), where models used information at the start or end of a long input better than information in the middle.
Anthropic's engineering post on context engineering from September 2025 describes the standard countermeasures: compaction (summarizing a conversation as it nears the limit), clearing old tool results the agent no longer needs, structured notes that the agent writes outside the context window and reads back later, and sub-agents that work in a clean context and return a short summary. All of them reduce tokens per turn, and they share one cost interaction worth planning for: rewriting earlier history invalidates the cache from the edit point onward. So trim at the source where you can, by making tools return the 20 rows that matter instead of 2,000, and compact at phase boundaries rather than every turn, so that one cache miss buys many cheaper turns afterwards.
The other habit that pays off is reading actual prompts. Log the full request at each step for a sample of tasks and look at one from late in a long run. You'll usually find the original goal buried under accumulated tool output, which is both the cost problem and the quality problem in one picture. A short block near the end of the prompt that restates the goal and current state, rebuilt each turn, is cheap and keeps the important part where models attend to it best.
05
route by task, escalate on failure
Most calls in an agent aren't hard. Classifying a ticket, picking a queue or pulling three fields out of an email doesn't need the model you use for drafting a careful reply, and the price gap is large: Claude Haiku 5.5 lists at $0.10 per million input tokens against Sonnet 5.5's $2, a factor of 20. A router doesn't need to be clever. Map each task to an ordered list of models, validate the cheap model's output, and escalate only when validation fails.
With a stub model that answers "refunds" (not a valid queue) on the small model for one input, the first call stays on Haiku and the second escalates, which is the behaviour you want to see in your logs:
Validation only catches malformed or out-of-range answers, not plausible wrong ones, so the routing table has to come from evals: run each task's test set on the cheaper model and move the task down only where it passes. And anything that doesn't need an immediate answer can go to the batch APIs at half price on top of routing.
06
metering cost per feature
You can't tell which of these levers matters until cost is visible per feature and per request, next to the code that caused it. Record every call with its token breakdown and compute the cost at insert time from a price table with validity dates, so a price change doesn't rewrite last month's numbers. Use the provider's request ID as the primary key, so a retried write is a no-op instead of a double charge.
Recording the same request twice inserts one row, and a daily query per feature shows spend next to the cache hit ratio, which is the first number to check when cost per task jumps:
Two details trip people up. Anthropic's input_tokens excludes cached tokens, so the cache read and write counts go in their own columns, while OpenAI reports cached_tokens as a detail inside input_tokens, so you subtract it before storing the uncached count. And Haiku 5.5 charges higher rates when a prompt exceeds 100,000 tokens, which a single price row can't express, so add a tier column if you send prompts that long. For traces, the OpenTelemetry GenAI semantic conventions (still marked development) define gen_ai.request.model, gen_ai.response.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.usage.cache_read.input_tokens and gen_ai.usage.cache_write.input_tokens, which most tracing backends can aggregate by feature. Keep per-account breakdowns in the table rather than as metric labels, since an account ID label on a Prometheus counter creates one time series per customer.
If you also bill customers for AI usage, this table is the source of truth for that too, with the caveat that what you charge should follow the value the customer gets rather than your raw token cost. Otherwise every caching or routing improvement you ship lowers your own revenue, which is the wrong incentive to build into a pricing model.
