Why are my LLM API costs so high, and how do I cut them?

Why are my LLM API costs so high, and how do I cut them? In most apps I review, input tokens cause most of the bill. The app sends the same long prefix on every call. The prefix has a system prompt, tool definitions and chat history. The app also sends every request to the strongest model. A cheap model can do much of that work. To cut the bill, do four things. Cache the repeated prefix. Move work that can wait to a batch API. Route easy requests to a small model. Measure cost per task, so you can see which change helps.

Why are LLM API costs so high? The short answer

I work as a fractional AI CTO. I look at LLM bills for startups and mid-size teams. The first call is nearly always the same. The team says the price per token is too high. I ask for a log of tokens per request. After that, we find that a small number of calls make most of the cost. Those calls are large, repeated and often not urgent.

This guide follows the order that I use in those reviews. First I look at where the tokens go. Then I use the cheapest fixes. The risky fixes, such as model routing, come last.

Where does the LLM spend go in an agent?

A chat app sends one prompt and gets one answer. An agent runs a loop. In each turn, the app sends the full context to the model again. The context has the system prompt, the tool definitions, the earlier messages and the tool results. Each turn adds more text. Thus the cost of turn 20 is much higher than the cost of turn 1.

I describe this loop in How to Learn AI Agents: Build the Harness. The harness controls what goes into the context on each turn. If the harness is careless, the bill grows with every step. In How Much Does It Cost to Build an AI Agent?, I say that a thin harness costs little because you pay mainly for tokens. A large harness adds engineering cost. It also gives you the tools to control the token cost.

Tool definitions are the first large item. Each tool has a name, a description and a JSON schema. The model reads all of them on every turn. Anthropic wrote about this in Code execution with MCP. They say that “tool descriptions occupy more context window space, increasing response time and costs”. They also say that agents connected to thousands of tools “need to process hundreds of thousands of tokens before reading a request”.

The second large item is tool results. Every result goes back through the model. In the same Anthropic post, a worked example moves a document between two systems. The task uses 150,000 tokens with direct tool calls. When the agent writes code to do the same work, it uses 2,000 tokens. Anthropic reports “a time and cost saving of 98.7%”. This is a vendor example, but the mechanism is general.

How much do tool definitions cost on every turn?

The cost depends on how many tools you connect and how large each schema is. The figure below lets you test this. I wrote about the same problem in MCP vs CLI: The Abstraction Tax.

The figure compares two designs. In the first design, the agent loads every tool schema at the start. In the second design, the agent sees a short entrypoint of about 80 tokens. It reads the help text for a tool only when it needs the tool. You choose the number of tools with the slider. You also choose the size of one schema: 550, 900 or 1,400 tokens.

Move the slider to 100 tools and watch the first bar pass 100,000 tokens. The second bar stays near the floor. Then change the schema size and see how fast the first bar grows. The numbers in the figure are my assumptions for a typical setup. Count your own tokens before you make a decision.

You pay this cost on each turn, before the user task starts. Take a 30-turn agent run that begins with 30,000 tokens of tool schemas. It sends 900,000 input tokens for the schemas alone. Prompt caching, which I describe next, reduces the price of this repeated prefix. It does not remove the tokens from the context window. So the best fix is to load fewer tools.

How much do prompt caching and batch APIs save?

Both providers sell discounts for work that is repeated or not urgent. The numbers below come from the official pricing and guide pages in October 2026. Check those pages before you plan a budget, because prices change.

The Anthropic figures are on the prompt caching page and the batch processing page. The page says that “Cache read tokens are 0.1 times the base input tokens price”. It lists lower multipliers for some models. For example, it says that cache hits on Claude Opus 5.5 and Claude Sonnet 5.5 cost 0.05 times the base input price. The batch page says the Message Batches API reduces “costs by 50%”.

The OpenAI figures are on the prompt caching guide and the Batch API guide. Caching is on by default for supported models. The guide also warns that it “does not guarantee that every request reuses stored KV tensors or has lower latency”. The Batch API gives “50% cost discount compared to synchronous APIs” with a 24-hour window.

Caching works best when many calls share a long, stable prefix. Agents, RAG apps with a fixed instruction block, and chat with long history all fit. The cache also expires. Anthropic uses a 5-minute or 1-hour lifetime. OpenAI keeps entries for a time that depends on the model. If your traffic is rare, the entry expires before the next call, and you pay the write price for no gain.

Batch APIs fit work that nobody waits for. Examples are evals, nightly summaries, data labeling, embeddings backfills and document extraction. A user who waits in a chat window cannot use a batch. Move the background jobs first. They often make up a large share of the token volume, and the change is small: you replace one call with a queued job.

When should you route to a smaller model?

Routing means that a cheap model sees the request first. A stronger model sees it only when the cheap one is not sure. The saving can be large, because most traffic is easy. But routing is also where most of the quality risk comes from. So I do it after caching and batching.

I use a simple process. Take 200 to 500 real requests from the logs. Label the correct answer, or write a clear check for it. Run the cheap model on all of them. Compare the pass rate with the strong model. If the gap is small for a task type, send that type to the cheap model. If the gap is large, keep the strong model.

Then add an escalation rule. The cheap model returns an answer and a confidence signal. The signal can be a log probability, a schema check, a self-rating or a second small verifier. When the signal is low, send the request to the strong model. The user waits a little longer on those requests, and the rest are cheap.

I built the same shape for safety checks, which I describe in How to Build LLM Guardrails. A cheap tier sees every request. Only a share of requests goes to the next tier. The last tier is an LLM judge, and it sees only a small fraction of traffic. The figure below shows how the bill changes with that share.

The figure uses toy units: 1 for tier 1, 10 for tier 2 and 100 for tier 3. These are not real prices. Slide “to encoder” down from 100% and watch the relative cost per request fall. Tier 1 still sees all traffic. The judge share stays small on purpose. The lesson for LLM routing is the same. The cost depends mostly on how many requests reach the expensive tier, so measure that share first.

What other fixes cut token use?

After the large levers, a few small changes help. Each one is easy to test.

In an agent, input volume is usually much larger than output volume, because the full context goes back on every turn. For this reason, I look at input first. I look at output when the product produces long text, such as reports or code.

Response caching is different from prompt caching. Prompt caching lowers the price of input at the provider. Response caching skips the model call completely. It fits FAQ-style traffic and repeated lookups. It does not fit personal or time-sensitive answers. Include the user scope and a freshness limit in the cache key, so that one user never sees the answer for another user.

Be careful when you trim context. If you cut too much, the agent forgets a fact and repeats a tool call. Test every cut with an eval set. Watch the turns per task as well as the tokens per turn.

How do you measure LLM cost per task?

You cannot cut a cost that you cannot see. The provider dashboard shows the total by day and by model. That is not enough. You need cost per task, per feature and per user. Without these numbers, a good change and a bad change look the same.

Log these fields for every model call: model name, input tokens, cached input tokens, output tokens and latency. Add an ID for the task, the feature and the user. Every provider returns the token counts in the response. Store them in your own database or a tracing tool. Add the price per token from the provider’s pricing page, and calculate the cost at read time. Then the same log works when prices change.

Next, group the calls by task. A task is one user goal, such as “answer a support ticket”. It may include 15 model calls. Cost per task is the sum of those calls. Compare this number with the value of the task.

Also track the cache hit rate. This is the share of input tokens that the provider served from cache. If you add caching and the rate stays near zero, the prefix changes between calls. Find the part of the prompt that varies and move it to the end.

Finally, track quality next to cost. A cheaper model that fails more tasks is not cheaper. Count the retries, the escalations and the human corrections as part of the cost. This is the same method that I use when I price a full agent build: count the whole system, not one line item.

Can you predict how much you will save?

I do not know your numbers, so I cannot promise a saving. In the reviews I do, the first two weeks give the largest change, because caching and batching need little code. I could not find a neutral, current study that gives an average saving across many teams. Treat any single percentage that you read online with care, including the ones on SEO pages that copy each other. Measure your own before and after.

One note on price changes. Providers cut prices often, and new models change the cost of each task. A routing choice that was right in spring can be wrong in autumn. Keep the eval set and the cost log, and run both again when a new model ships.

So, why are my LLM API costs so high, and how do I cut them?

Your costs are high because the app sends large, repeated input to a strong model for every request, and nobody measures cost per task. To cut them, log tokens per task first. Then cache the stable prefix and move background work to a batch API at 50% off. Load fewer tools. Route easy requests to a smaller model and escalate the hard ones. Check quality after each step. If you want help with a cost review of your agent, I do this work as a fractional AI CTO.