Your AI assistant keeps eating the same prompt twice

The cost of repeated context

A customer service assistant receives the same returns policy, product information and conversation instructions with every question. Only the customer's specific query changes. To the team, it looks like one ongoing conversation. To the AI model, the lengthy background is part of the input each time, unless the provider can reuse it.

That is what prompt caching is designed to address. Put simply, a section of the prompt that has already been processed is kept available for future requests. The new question is added, and the answer is still generated separately. How a provider stores the cache, how long it remains valid and what counts as a cache hit all depend on the model and API.

So the financial benefit does not depend on the length of your documents alone. What matters is how often the same context actually comes up, and how much the provider charges for cached input. A cache that rarely gets hit may sound useful on paper and make little difference to your bill.

What five cents means in this example

Epoch works through a hypothetical Opus-5 API example: 100,000 tokens of cached context, 1,000 new input tokens and 1,000 output tokens. Tokens are the units of text many AI providers use to bill for usage. In this example, the cached input costs US$0.05.

There is also a US$0.00625 cache write cost, based on the five-minute cache duration described by Epoch. The 1,000 output tokens cost US$0.025. That makes US$0.08125 for the hypothetical request in total. This is a worked example, not a standard rate or a guarantee of current prices.

The figures still show where it is worth looking: reused context can cost less than the same amount of ordinary input. But caching does not make the whole request free. New inputs, generated answers and, where applicable, writing to the cache remain part of the bill. What happens when your context is much longer?

Long prompts still take time

If you have ever waited for an assistant's first answer, you will know the difference between a quick response and a conversation that takes several seconds to get going. The time to the first token is often called TTFT. A cache can change the cost of reused context, but the available measurements do not show a fixed time saving for every cached request.

Epoch compares TTFT for four models across different context lengths. In its measurements, GPT-5.6 Terra and Sol showed a much steeper increase in TTFT with longer prompts. Claude Sonnet 5 stayed closer to a linear scaling pattern. These results depend on the models tested. They are not a universal ranking of AI services.

The technical concern is easy to understand: with full attention, each token takes the whole context into account. According to Epoch, the computational workload grows quadratically with context length. For businesses, that means it is worth measuring the time to the first answer on long prompts, even if a provider charges less for cached context.

Caching depends on repetition

If each department sends different rules, documents and instructions, less may be repeated than you think. The long context may still be costly or slow, even if the word “caching” sounds like an easy saving.

It is more useful to look at how work actually gets done. Which instructions appear in every request? Which product or process details are the same across many requests? And which information changes daily? This helps you separate stable shared context from the customer's specific question. Check the provider's rules to see whether small changes can prevent a cache hit.

Cache duration matters too. In the Epoch example, a five-minute cache write is included in the calculation. If there are long gaps between requests, reuse may work differently than it does for a heavily used assistant. One successful test will not tell you much about a typical month.

Measure before changing your prompt

For a small business, start with a focused test. Choose an assistant whose requests regularly include the same information. Over a representative period, record the number of requests, the length of the repeated context, the cost per request and the time to the first answer. Then compare similar requests with and without caching, if your platform allows it.

For a mid-sized business, results may vary by use case and model. An internal assistant using stable HR policies is different from a sales assistant reading changing offers. In a larger business, break the figures down by agent, model and request as well. OpenRouter's Activity dashboard and Analytics API include measures such as cache hit rate and average cost per million tokens.

Check the billing rules for the provider you use. Epoch describes a special pricing tier for OpenAI requests with more than 272,000 input tokens: the standard input and cached input prices are doubled, and the standard output price is multiplied by 1.5. Thresholds and rates can change. Check your provider's current documentation before testing.

• Start with one use case and a clearly defined prompt.
• Track cache hits, costs and TTFT together.
• Compare recurring requests across several typical working days.
• Decide who will regularly check price changes and cache rules.

What this means for your business

Your first step does not have to be a major overhaul of your AI setup. Take the latest recurring requests for one assistant and mark the text that stays the same each time. If they contain lengthy policies or product information, they are a good candidate for a controlled caching test. If almost everything changes from one request to the next, the bigger opportunity may lie elsewhere.

Decide in advance what success looks like for your team. A lower bill is not much help if answers arrive too late or miss important information. On the other hand, a modest saving on token costs may be worthwhile if the assistant handles many recurring tasks. Make the call based on cost, response time and reliability together.

Picture next Monday: you can see which requests reuse shared context, which still start with long prompts and what each group costs. Then you can decide whether to expand caching, shorten the prompt or structure the request differently. That evidence is what turns a technical feature into a business decision.

Frequently asked questions

More articles

• Practice & mistakesThe AI agent does not just answer. It can act wrongly
• Practice & mistakesYour AI cites sources and can still give the wrong answer
• Strategy & trendsThe AI IQ trap: A high score won’t stop costly mistakes