Where the AI money actually goes.

The Pooled Key Trap: Everyone spends, nobody owns.

Here is how almost every company starts with AI: a platform team sets up an enterprise account with OpenAI or Azure, creates a shared internal proxy, gives out an API key, and tells teams to go build things.

For a few weeks, everyone is happy. Teams build customer support bots, internal wiki searchers, and drafting tools. Then the invoice arrives. Finance looks at a single line item for $38,000 and asks: who spent this? What did we get for it? Did this help customer success or did an engineer accidentally leave an evaluation script running all weekend?

Nobody knows, because the proxy just pooled all the traffic together.

Microsoft's paper on the economics of agent optimization touches on something very sensible here: organizations have to operate at three different speeds:

• In the moment (Runtime): Don't use a massive frontier model to classify a simple email into three categories. Route basic jobs to fast, cheap models, and save the big models for hard reasoning. Cache static headers so you don't pay to parse the same system prompt over and over.
• Over weeks (System design): Look at what tools the agents are actually using. If an agent has access to 25 different tools, its prompt is clogged with tool descriptions it never calls. Trim the toolbox.
• Continuously (Governance): Put real limits on company accounts that stop spend before it happens, not after finance sees the invoice.

In my own work, my rule of thumb is straightforward: Never track prompt counts. Track accepted outcomes.

The Local Cap Trap: Safe agents that add up to a disaster.

A very common mistake when writing agent code is putting limits only on the individual model calls. You write in your config: "Never let a single call use more than 2,000 tokens or cost more than five cents." You test it, see that every call is cheap, and feel responsible.

Then you build a multi-agent workflow. Say you have a lead coordinator that assigns tasks to three helpers: a research agent, a writer agent, and a fact checker. Each helper is strictly capped at five cents per step.

The research agent looks up a topic, gets an unexpected search result, and tries again. It makes 20 calls. Total: $1.00. The writer agent drafts three variations because the outline was vague. Total: $1.20. The fact checker flags a minor phrasing dispute and asks the researcher to check again. Total: $1.50. Every single step was completely compliant with your five-cent rule. Yet the complete task just cost $3.70 instead of fifty cents.

Budget for one report: $0.75
┌─────────────────────────────────────────────────────────┐
│ Coordinator Agent                                       │
├───────────────────┬───────────────────┬─────────────────┤
│ Researcher        │ Writer            │ Fact Checker    │
│ Cap: $0.05/call   │ Cap: $0.05/call   │ Cap: $0.05/call │
│ Made 25 calls     │ Made 22 calls     │ Made 18 calls   │
│ Spent: $1.25      │ Spent: $1.10      │ Spent: $0.90    │
└───────────────────┴───────────────────┴─────────────────┘
Total Task Spend: $3.25 (Over 400% of budget)
Individual rule violations: Exactly zero.

Standard monitoring tools like Datadog or Langfuse are great for diagnostics, but they are spectators. They tell you: "Hey, 15 minutes ago an agent spent $150 in a circular loop." That's nice to know, but the money is already gone.

This is why projects like TokenOps talk about in-path admission control. You don't let agents hold their own little budgets. You give the whole job one run ID and one shared ledger. Before an agent opens a network connection to the model, it must check the shared balance. If the remaining budget cannot cover the worst-case output of that next call, the gate shuts immediately. It either cuts the output short, forces the agent to summarize, or stops the run before the provider charges your credit card.

The Debug Tax: Why fixing agent bugs burns a hole in your pocket.

In standard programming, debugging costs nothing. When Python throws a KeyError or a Java service crashes, you don't pay a fee. You look at the line number in the stack trace, open your editor, write a unit test, fix the bug, and run the test locally. Compute cost: zero dollars and zero cents.

With AI agents, debugging is an active bill. Imagine an agent that reads a 50-page vendor contract, extracts forty key clauses, checks them against your compliance rules, and creates an intake ticket. On Step 12, it encounters an unusual date format and throws an unhandled error.

What does a developer usually do? They tweak the prompt for Step 12, and then run the whole script again from the beginning. They pay for Steps 1 through 11 all over again just to reach Step 12. If the bug doesn't trigger because the model sampled slightly different words, they run it again. By the time they fix that single date parsing issue, they've spent $25 in live model calls just looking at the bug.

Chronicle tackles this through boundary recording and replay. It is an idea taken from good distributed systems practice: treat everything that crosses a network boundary (a model response, a tool call, a database read) as an immutable record.

When something fails at Step 12, you don't run the script live again. You run it against the recorded cache of Steps 1 through 11. That replay is instantaneous and makes zero model calls. You only pay to execute Step 12. Once you verify your prompt tweak works, you save that recorded session as a regression test so future prompt changes don't silently break it later.

Context Bloat: You are paying rent on every single word.

When teams discover that models respond well to clear instructions, they tend to overdo it. They create an AGENTS.md or RULES.txt file in their repo and dump everything in there: team coding conventions, company values, API design guidelines, database schemas, and twenty paragraphs of formatting advice.

It feels organized. But it ignores how LLMs work.

Models don't have long term memory while running a conversation. Every single time the user asks a question, or the agent runs a tool, the entire history is bundled up and sent back to the model. If your system context is 8,000 tokens long, and your agent takes 15 turns to complete a task, you didn't buy 8,000 tokens of context. You bought 120,000 tokens of context ($8,000 \times 15$). Most of that money was spent re-reading instructions the model didn't even need for that specific step.

Researchers at ETH Zurich actually tested this systematically in their paper evaluating AGENTS.md files across 138 software tasks and 12 codebases. What did they find?

• No real boost in success: Loading repository context files didn't noticeably increase how often agents solved the actual problem.
• A 20% higher bill: The token cost jumped by more than 20% across every model they tested.
• The Compliance Trap: When you give an agent a massive list of rules, it wastes its reasoning budget trying to satisfy formatting nitpicks instead of focusing on the actual problem. It wanders around making sure its indentation matches paragraph 4 instead of fixing the logic error in your code.

The lesson here is simple: Context is an expensive liability. Keep your base prompt tiny. Only include rules the model cannot possibly figure out on its own. If you have detailed API documentation, don't stuff it into the system prompt; let the agent look it up via a tool only when it actually needs it.

Too Much Process: When the scaffolding costs more than the work.

Because agents can be unpredictable, the software world has swung hard toward structure. We now have elaborate planning frameworks: Specification-Driven Development, multi-step constitution checks, and multi-agent debate protocols.

Some structure is great. But process itself is an inference workload.

Jamie Telin did a very clean, real-world comparison of this by running the exact same programming task through two different frameworks: GitHub Spec Kit and Fission AI's OpenSpec. Both tools got the job done. But look at what they spent:

Why did Spec Kit burn roughly double the tokens? Because it had more conversational ceremony. It had the model write a specification, critique the specification, validate it against rules, generate an intermediate task checklist, update five markdown files, and talk through its thoughts at every step.

If you are modifying core flight control software or high frequency trading systems, spending 2x the tokens on rigorous checks is cheap insurance. But if you are building an internal admin dashboard or writing a migration script, that extra ceremony is just pure waste. You are paying the model to fill out paperwork for itself.

Five rules I use to keep AI projects honest.

Putting all this together gives me five practical rules whenever someone brings an AI project to my desk:

1. 01
   Kill shared API keys: Give every team and workflow its own tag. If a workflow can't prove what business outcome it produced, don't give it a budget.
2. 02
   Budget the whole task, not the step: Put one hard limit on the complete run across all helpers. Stop runaway loops before the network socket opens, not after an alert pings Slack.
3. 03
   Don't pay to debug: Record your tool and model boundaries. When an agent breaks on step 10, replay the first nine steps from disk for free instead of paying the provider again.
4. 04
   Starve your prompts: Strip out generic advice like "be helpful" or "write clean code." Put your system prompts on a strict diet. Every word you add is rent you pay on every single turn.
5. 05
   Watch for paperwork loops: Measure how many tokens your agents spend planning versus doing. If a framework takes forty turns before it edits a single file, question whether that process is actually helping.

Tokens are not server hours.

We are going through the same growing pains with AI that we went through with cloud hosting fifteen years ago. Back then, teams left oversized virtual machines running over the weekend because nobody was looking at the bill. Eventually, we learned how to size instances and turn them off when idle.

With AI, the waste isn't idle servers. It's wandering prompts, unbudgeted agent loops, bloated instructions, and debugging reruns. Good FinOps isn't about refusing to use the best models. It's about designing your systems so that when you do spend money on a model, it's actually doing useful work for someone.