Start with a cost ledger
A large token count does not identify the expensive part of a workflow. Input and output have different prices. Cache creation may cost extra. A cheap call repeated ten times can cost more than one successful call on a larger model.
Record model version, input tokens, cached tokens, output tokens, tool calls, retries, latency, and the final review outcome for every run. Keep content out of the metrics record unless the retention policy explicitly permits it.
Calculate cost from disjoint billing categories: uncached input × its rate, cache creation × its rate, cache reads × their rate, and output × its rate. Add tool charges.
Use the provider's usage definitions to avoid counting cached input twice.
Why a growing conversation gets expensive
Suppose each turn adds d tokens and every request resends the complete history. Across n requests, the added input is d × n × (n + 1) / 2. A fixed system prefix of P tokens adds n × P.
For ten turns adding 1,000 tokens each, the growing history contributes 55,000 input tokens. A 2,000-token system prefix adds another 20,000. These are illustrative counts before cache discounts, not a dollar estimate.
The cumulative input grows quadratically under those assumptions. Each individual request grows linearly. Truncation, summarization, state handling, and prefix reuse change the bill; repeated context need not be recomputed from scratch.
Calculate when caching pays
OpenAI's prompt-caching guide and Anthropic's documentation describe reuse of matching prompt prefixes. Put stable instructions and tool definitions first.
Put changing questions, timestamps, and retrieved material after the reusable boundary.
Let P be prefix length, N the number of uses, r the ordinary input rate, w the cache-write rate, and c the cache-read rate. Without caching, that prefix costs N × P × r. With one write, it costs P × w + (N − 1) × P × c.
For a hypothetical write rate of 1.25r and read rate of 0.1r, two uses cost 1.35Pr instead of 2Pr. The saving applies to that prefix, not the whole workflow. Expiry or a changed prefix can require another write.
Check the current model's rates, minimum eligible prefix, retention setting, and observed hit count. A configured cache with no hits is an added cost, not an optimization.
Bound the work and preserve the evidence
Put tool-iteration, elapsed-time, and spend limits in application code. When the same failed operation repeats without new information, stop or escalate. The model should not be able to raise its own limit.
Replace bulky tool results with durable references and the fields required for the next step. Preserve the original evidence separately under the retention policy. A summary should not silently remove a constraint needed to finish the task.
Route a task to a smaller model only after testing it against the same acceptance criteria. Count repair calls, review time, and failures. Lower cost per call can coexist with higher cost per completed job.
Run one controlled comparison
Replay a representative set of workflows with the current setup and one proposed change. Hold task inputs and acceptance rules constant. Report total spend, accepted results, correction rate, and latency for each setup.
Cost per accepted result = total run cost ÷ accepted results. If a cheaper configuration produces no acceptable results, it has not become economical. There is no defensible universal savings percentage without this measurement.
Frequently asked questions
Does input always dominate an API bill?
No. The result depends on token volumes, model-specific rates, cached input, output length, retries, and tool charges. Measure each category.
Does a cache hit make the whole request cheaper by the advertised discount?
No. A discount applies to eligible cached tokens. New input, output, cache writes, and tool charges remain separate.