On this page
Prompt caching reuses processing associated with an unchanged portion of a model request. It can reduce repeated input work when an agent sends the same instructions, tools or reference material across requests. Start by identifying a stable prefix, then measure actual cache usage and billed cost. Do not assume that every repeated request qualifies or that a cache hit means the answer itself was reused.
Claude's prompt-caching documentation and OpenAI's caching guide describe provider-specific behavior. Keep thresholds, retention and prices tied to the provider and model you actually use rather than transferring settings between them.
Separate caching from memory and saved answers
These features solve different problems. An answer cache returns a previously saved result when the application considers another question equivalent. Persistent memory stores information that a later task can retrieve. Prompt caching concerns repeated processing of eligible request input.
Consider a customer-support agent that repeatedly receives the same product manual and instructions. Its user question changes on each request. A reusable prefix may reduce input-processing cost while the model still generates a new answer for that question. The application must continue checking that the manual version is appropriate and that private context belongs to the current user.
Avoid treating caching as a factual freshness mechanism. If a product policy changes, the application needs to update the supplied material. Efficiently processing an outdated manual still produces a workflow with outdated evidence. Record the source version independently of cache status.
Put stable material before changing task data
Design the request so reusable material stays consistent. Place stable operating instructions, tool definitions and shared reference material before the task-specific information when that structure is compatible with the provider's documented behavior. Keep arbitrary timestamps and request IDs out of the stable portion when they are not needed there.
Do not move user-specific data into a shared prefix merely to increase reuse. Access scope and correctness come first. If each user has a different approved document set, preserve that distinction. A smaller eligible prefix can be more useful than a larger one that exposes the wrong information or binds the task to stale material.
| Request component | Typical role | Review question |
|---|---|---|
| Operating instructions | Stable guidance | Did the procedure or authority change? |
| Tool definitions | Available operations | Does this user have the described capabilities? |
| Shared documentation | Versioned evidence | Is this the correct current source version? |
| User context | Scoped task information | Is every record authorized for this user? |
| Current request | Changing objective | What result must this invocation produce? |
The table describes a request-design pattern, not a universal caching guarantee. Verify the provider's matching rules and response fields. Some requests will remain ineligible because they do not meet the actual model or service requirements.
Calculate savings from reported usage
Compare the cost of the same representative request set with and without the eligible caching behavior. Retain ordinary input, cache-write or cache-read usage where the provider exposes those categories. Track output usage separately. Changing the answer length while testing input caching can confuse the comparison.
For a simple illustrative model, define these symbols:
P = eligible prefix tokens
I = ordinary input price per token
W = cache-write price per token
R = cache-read price per token
n = eligible repeat reads after one cache write
uncached_prefix_cost = (n + 1) * P * I
cached_prefix_cost = P * W + n * P * R
prefix_saving = uncached_prefix_cost - cached_prefix_cost
This arithmetic excludes changing input, output, tool charges and other costs. It is a calculation template, not a quoted vendor price. Substitute the actual billing categories and prices for your deployment. If the provider uses a different charging model, adapt the formula instead of forcing the invoice into these symbols.
For a synthetic example, let the ordinary unit cost be 1, write cost 1.25 and read cost 0.1 for one unit of eligible prefix. With five repeat reads, the uncached cost is 6 units and the cached cost is 1.75 units. Those numbers demonstrate the calculation only. They are not current prices or measured savings from a production agent.
Investigate misses with controlled changes
When reported cache use is lower than expected, compare the actual requests. Look for changes in shared instructions, tool order, document content or identifiers inserted ahead of the reusable material. Confirm that the request meets the documented eligibility conditions and reaches the intended provider and model.
Change one variable at a time. First repeat an identical safe request. Then change the user task while keeping the eligible prefix stable. Finally change the reference version deliberately. Preserve the provider usage responses for each case. This gives you evidence about your own request construction without inferring behavior from a generic latency difference.
Keep timing results separate from billed usage. Network delays, model generation and external tools can dominate elapsed time. A cache hit may reduce one part of the request while the complete task still takes longer for another reason. Describe the part you measured and the uncertainty that remains.
Build a small cost-review workspace
Use Atoms' AI app builder to prototype a request-cost dashboard with synthetic usage rows. The app can help reviewers compare input categories, reference versions and verified outcomes. A dashboard should never invent missing provider usage or treat lower cost as proof of better answers.
Build a prompt-cost review app with synthetic request records.
Store provider, model, reference version, ordinary input, cache write,
cache read, output usage and the corresponding unit prices.
Keep unavailable usage fields unknown rather than replacing them with zero.
Calculate input-category costs and show the formula beside every total.
Allow comparisons only within the same model, prices and test dataset.
Show task-quality checks separately from cost and latency.
Support CSV import and export; do not connect live provider credentials.
Label all demonstration prices and outcomes as synthetic examples.
For the wider workflow, read the agent orchestration guide. Caching can reduce repeated processing, while good orchestration keeps the task focused and checks the result. Both need evidence from the actual deployment.
Use the sample brief to start a cache measurement dashboard in Atoms.
Conclusion
Keep a correct, authorized request prefix stable, then verify provider-reported cache usage and billing. Use controlled examples to explain misses and separate input savings from total task cost and quality. Prototype a cost-review app with Atoms if your team needs a clearer way to inspect those records.
Create a first version with synthetic data and review the results.
FAQs
Q1: Does prompt caching return an old answer?
Prompt caching concerns reuse of eligible input processing, while an answer cache returns a stored response. A provider can report a cached prefix and still generate a new answer for the current question. Your application should keep those concepts separate in both implementation and reporting. If you also cache answers, define the conditions under which a saved answer remains appropriate. Changes in source documents, access scope or task context may require a fresh answer even when part of the model input remains reusable.
Q2: How do I know whether caching saves money?
Use actual reported usage and the prices that apply to the same provider, model and billing period. Compare a representative request set while holding the task and output requirements constant. Include cache-write and cache-read costs where applicable, then add changing input, output and tool costs separately. Missing usage remains unknown. A synthetic break-even calculation helps explain the method, but production savings require invoice-compatible measurements. Lower latency alone does not establish lower billed cost, and a lower-cost run still needs to meet the task's quality criteria.
Q3: Should I add more text to reach a caching threshold?
Do not add irrelevant material solely to pursue cache eligibility. Extra context still has processing and maintenance costs and may obscure the information the task needs. First make the request correct and concise, then examine how useful shared material can remain stable. Check the provider's current eligibility requirements for the actual model. If the request is too small or changes too much to benefit, preserve a simple uncached workflow. Caching is an optimization of useful input, not a reason to inflate every request.
