Reducing AI Token Usage with Caching

AI applications can consume a surprising number of tokens. The problem is often not the user's question. It is everything the application keeps sending with it.

As developers add AI to applications, coding workflows, security tools, agents, and automation, token usage becomes something that needs to be treated as an architectural concern rather than simply an API billing detail. A single AI request may contain a system prompt, tool definitions, application rules, reference documents, source code, conversation history, retrieved data, previous tool results, and finally the user's actual request.

That context can become large very quickly.

If much of it remains unchanged between requests, repeatedly processing it is wasteful. Modern AI platforms now provide caching mechanisms specifically designed to reduce that repeated work. OpenAI, Anthropic Claude, Google Gemini, and xAI Grok all provide forms of prompt or context caching. The implementations differ, but the basic idea is the same:

Stable context should be reused rather than repeatedly processed from scratch.

Caching alone is not the complete answer, however. Efficient AI applications also need retrieval, application-level caching, conversation compaction, selective tool loading, and careful context construction. The goal is simple:

Send the model the smallest amount of high-quality context required to complete the task.

Where All the Tokens Come From

A basic AI request may look simple:

A production application usually looks more like this:

Every one of those components contributes tokens. The user's actual question may contain only twenty tokens, while the application sends another 20,000 tokens of supporting context around it. That may be completely justified for the first request. The problem appears when the next request contains almost exactly the same 20,000 tokens. And then the next one does too.

The Simplest Example

Imagine an application analyzing a large reference document. The document consumes approximately 20,000 tokens. The user asks a question:

Then the user asks another question:

If ten questions require the document to be processed repeatedly, the application may send roughly:

Most of those tokens represent information that never changed. This is exactly the type of workload prompt caching is intended to improve.

Prompt Caching Changes the Pattern

With caching, the large stable portion can be reused. Conceptually:

The model still receives the required context logically, but the provider can reuse previously processed prompt content rather than performing all of the same computation again. Depending on the provider and model, this can reduce input cost and improve latency. The important word is stable. Most prompt caching implementations depend heavily on repeated prompt prefixes.

Stable Context Should Come First

A useful way to think about a prompt is to divide it into two sections. The first is relatively static:

• System instructions
• Application policies
• Tool definitions
• Reference documentation
• Examples

The second changes constantly:

• Current task
• Latest tool response
• Current file
• User question
• Timestamp
• Request-specific values

A cache-friendly prompt therefore looks like:

This design is important because changing something near the beginning of a prompt can invalidate much more of the reusable prefix.

OpenAI Prompt Caching

OpenAI supports prompt caching for repeated prompt prefixes. For many supported models, caching happens automatically when enough of the beginning of a request matches previously processed content. Applications can also use mechanisms such as prompt_cache_key to help group related requests and improve cache utilization. The pattern is straightforward. Imagine an application repeatedly sending:

• SYSTEM INSTRUCTIONS
• APPLICATION RULES
• TOOL DEFINITIONS
• REFERENCE DOCUMENTATION
• USER REQUEST

If the first four sections remain the same, they form a strong reusable prefix. The next request becomes conceptually:

OpenAI exposes cached token information in API usage data, allowing developers to determine whether their prompt structure is actually benefiting from caching. This is particularly valuable for agentic applications because tool schemas and system instructions can represent a significant portion of every request. Instead of looking only at: Total input tokens it becomes useful to think about:

The more stable context you can reuse, the less repeated work is required.

Claude and Prompt Caching

Anthropic provides prompt caching for Claude and gives developers significant control over what should be cached. Claude can cache reusable portions of a request such as:

• system instructions;
• tool definitions;
• documents;
• messages;
• examples;
• other large pieces of stable context.

With explicit caching, developers can place cache controls around portions of the prompt that are expected to remain stable. Conceptually:

The stable material can then be reused by subsequent requests while the current investigation continues to change. This is especially useful for long-running agents. An agent may execute dozens of model calls while using the same:

• System instructions
• Tool definitions
• Security methodology
• Application architecture
• Reference documentation

There is little reason to repeatedly perform the full processing cost of that information if it has not changed. Claude also exposes cache-related usage information, which makes it possible to measure how many tokens are being written to or read from the prompt cache.

Grok and xAI Prompt Caching

xAI also supports prompt caching with Grok. Like the other providers, the effectiveness of caching depends heavily on maintaining a stable beginning to the prompt or conversation. A conversation might look like:

The next request should generally preserve the existing history and append new information rather than unnecessarily rewriting earlier messages. Conceptually:

Changing, deleting, or reordering early content can reduce the amount of context that can be reused. xAI also provides mechanisms to help related requests reach infrastructure where the appropriate cached context is more likely to exist. The larger architectural lesson applies regardless of provider:

If content is intended to be reused, keep it stable.

Gemini Context Caching

Google Gemini supports both implicit and explicit forms of context caching. Implicit caching allows repeated prompt content to benefit automatically when the request structure and model support it. Explicit caching goes further by allowing an application to create reusable cached context and reference it in later requests. This can be particularly useful for large inputs such as:

• technical documentation;
• large PDFs;
• videos;
• source-code collections;
• product catalogs;
• research material;
• large system instructions.

The architecture might look like:

Instead of repeatedly submitting the same reference material, subsequent requests can reuse that context. Gemini's explicit approach can be attractive for applications where a known dataset will be queried repeatedly over a defined period.

The Providers Differ, but the Principle Is the Same

OpenAI, Claude, Gemini, and Grok do not implement caching identically. The details vary around:

• automatic versus explicit caching;
• minimum cacheable prompt sizes;
• cache lifetimes;
• cache routing;
• pricing;
• retention;
• API parameters;
• usage reporting.

Those implementation details change over time and should always be checked against the provider's current documentation.

Architecturally, however, they are solving the same problem:

Applications repeatedly use the same context. Caching provides a way to stop treating every request as if the model has never seen that context before.

Caching Matters Even More for Coding Agents

Coding agents are one of the best examples of how token consumption can grow. A coding assistant may need context containing:

• System instructions
• Repository architecture
• Coding conventions
• Tool definitions
• Package information
• Source files
• Previous modifications
• Conversation history
• Test output
• Current task

The agent may then perform a workflow such as:

Each step may generate another model request. Suppose each request contains 30,000 input tokens. Twenty requests could theoretically result in:

30,000 x 20

= 600,000 input tokens

Yet perhaps 20,000 of those 30,000 tokens contain the same instructions, repository information, and tool definitions. That is where caching starts to matter significantly. The agent is doing new work on every turn, but much of the information surrounding that work has not changed.

Prompt Caching Is Not the Same as Application Caching

Provider-side prompt caching solves only one part of the problem. Applications should also cache their own data. Imagine an AI security application retrieving an inventory of 8,000 assets.

Without application caching:

Another agent asks for the same inventory thirty seconds later following the same pattern. There are two separate inefficiencies. First, the application repeatedly queries the backend. Second, it may repeatedly send the same dataset to the model. A better architecture is:

The application can cache the expensive result and then provide only the portion required for the current task.

Cache Data Outside the Model Too

There are many useful candidates for application-level caching.

These might include:

• API responses;
• database queries;
• DNS results;
• parsed documents;
• repository metadata;
• file indexes;
• embeddings;
• configuration data;
• asset inventories;
• cloud resource information;
• tool outputs;
• summarized datasets.

For example:

If two agents need the same relatively stable information, there is little benefit in retrieving and processing it twice. Caching at the application layer can reduce backend activity before token usage is even considered.

Cache the Data Before You Cache the Prompt

This distinction is important. Suppose your application retrieves 100,000 security events and places all of them into a cached prompt Yes, prompt caching may reduce repeated processing cost. But you are still carrying 100,000 events inside the model's context. The first question should therefore be:

Does the model need all of this information?

Imagine the user asks: Show me failed authentication events for Maya. The inefficient design is:

A better design is:

The database is better at filtering records. The language model is better used to interpret the resulting evidence. This is an important distinction when trying to reduce token consumption. Caching does not make unnecessary context necessary.

Retrieval Can Save More Tokens Than Caching

This is one reason Retrieval-Augmented Generation, or RAG, remains so useful. Suppose an application's knowledge base contains one million tokens. The user asks a question that requires only a few sections of one document. Sending the entire knowledge base would look like:

Retrieval changes the architecture:

Caching and retrieval solve different problems.

• Retrieval reduces how much context you need.
• Caching reduces the repeated processing of context that still needs to be there.

Used together, they are far more effective than either technique alone.

Do Not Send Entire Datasets Unless You Need Them

Large datasets are particularly easy to misuse with AI. Consider an application containing:

• 100,000 users
• 500,000 authentication events
• 50,000 devices
• 20,000 applications

A request asks: Which disabled users successfully authenticated during the last 24 hours?

The model does not need every user and authentication event. Traditional application logic should perform the initial reduction:

This may turn hundreds of thousands of records into a handful. That can save dramatically more tokens than prompt caching alone.

Cache Tool Results

AI agents frequently call tools. Those tools may query:

• APIs;
• databases;
• DNS;
• cloud platforms;
• source repositories;
• asset inventories;
• configuration services;
• security platforms;
• documentation systems.

Some of those results change constantly. Others do not. If an agent retrieves something that will remain valid for a reasonable period, cache it. For example:

An application might define different cache lifetimes depending on the information:

Information Type	Cache Lifetime
DNS result	5 minutes
Application configuration	15 minutes
Asset inventory	30 minutes
Repository metadata	30 minutes
Documentation index	24 hours
Static reference data	Several days

Those durations are examples rather than rules. The correct time-to-live depends on how quickly the underlying information changes and how stale the application can safely allow it to become.

Long Conversations Need Compaction

Conversation history is another easy way to consume tokens unnecessarily. A long-running assistant or agent might eventually accumulate:

• Turn 1
• Turn 2
• Turn 3
• Turn 4
• ...
• Turn 67
• Turn 68
• Turn 69
• Turn 70

If all seventy turns are sent with every request, the conversation becomes progressively more expensive. Eventually, old details may also begin competing with current information for the model's attention. A better strategy is periodic compaction:

The summary might preserve:

• Current objective
• Important decisions
• Known facts
• Files changed
• Constraints
• Outstanding tasks
• Current errors
• Relevant identifiers

The original conversation can remain stored externally. It simply does not need to be placed into every model request.

Summaries Should Preserve State, Not Conversation

There is an important difference between summarizing a conversation and preserving application state. A weak summary might say:

The user and assistant discussed the application,
looked at several errors, and made some changes.

That is not very useful. A state-oriented summary would instead preserve:

Objective:
Reduce authentication latency.

Current implementation:
OAuth 2.0 authorization code flow.

Files modified:
auth.ts
token-cache.ts

Known issue:
Refresh tokens are not being reused.

Decision:
Move token persistence into Redis.

Next step:
Update TokenProvider.getToken().

That is far more useful to the next model call and usually consumes fewer tokens than replaying the entire conversation.

Tool Definitions Are Tokens Too

Tool definitions are easy to overlook. An agent with access to fifty tools may receive schemas containing:

• Tool name
• Tool description
• Parameters
• Parameter descriptions
• Enums
• Nested objects
• Return types
• Examples

A large tool catalog can consume thousands of tokens before the user has asked anything. Prompt caching can help because tool definitions are often stable. But there is an even better optimization. Do not expose tools the agent does not need. Instead of:

consider:

This reduces token usage while also reducing the number of choices the model has to reason about. It is both a cost optimization and an agent-design improvement.

Small Changes Can Destroy Cache Efficiency

Consider this prompt:

Timestamp: 2026-09-10T15:30:41
Request ID: 73B18
Session ID: 841A2

System instructions...
Application rules...
Reference documentation...
Tool definitions...

The request begins with values that change every time. That is a poor structure for prefix caching. Instead:

System instructions...
Application rules...
Reference documentation...
Tool definitions...

Timestamp: 2026-09-10T15:30:41
Request ID: 73B18
Session ID: 841A2
Current request...

The large reusable material remains stable at the beginning. The volatile values move toward the end. The model receives the same information, but the prompt is far more cache-friendly.

Think About Prompt Structure as an API Contract

Once caching becomes important, the structure of the prompt itself should become deliberate. For example:

1. System instructions
2. Security and application rules
3. Tool definitions
4. Stable reference information
5. Long-lived examples
6. Current conversation state
7. Latest tool results
8. Current user request

The first portion changes rarely. The last portion changes frequently. That provides a natural caching boundary. It also makes the application easier to debug because developers can identify exactly which part of the context changed between requests.

Output Tokens Matter Too

Reducing token usage should not focus only on input. Many applications ask models to produce long explanations even when the output will immediately be consumed by another piece of software. For example:

Analyze these records and provide a detailed explanation of
everything you found, including your reasoning and suggested
next actions.

That may generate hundreds or thousands of output tokens. If another application component needs only a decision, request structured output instead:

{
  "status": "pass",
  "finding": "",
  "confidence": 0.96,
  "next_action": "none"
}

Human-readable explanation can always be generated later when someone actually needs it. This leads to another useful rule:

Generate prose for people. Generate structure for software.

Structured Output Can Reduce Agent-to-Agent Costs

This becomes even more important when multiple AI agents communicate with each other. Imagine one agent produces:

I reviewed the server and found several interesting things.
The system appears to expose TCP 443 and it looks like the
certificate contains a hostname that might be useful for
further investigation...

Another agent then needs to parse that prose. A better handoff might be:

{
  "host": "10.20.30.15",
  "port": 443,
  "protocol": "https",
  "hostname": "portal.example.local",
  "confidence": 0.98
}

That response is:

• shorter;
• easier to cache;
• easier to validate;
• easier for software to consume;
• easier to pass between agents.

Agent-to-agent communication does not need to resemble a human conversation.

Use Traditional Code Whenever Traditional Code Is Better

One of the easiest ways to reduce token usage is not to call an AI model at all. Developers sometimes send work to an LLM that could be handled more efficiently by normal application logic. For example:

The model should be used where reasoning, language understanding, interpretation, synthesis, or decision support provides value. It does not need to become the database engine, calculator, cache, parser, and search system at the same time.

Choose the Right Model for the Task

Model choice also affects application efficiency. Not every operation requires the most capable reasoning model available. A multi-model architecture might look like:

A smaller model may be perfectly adequate for classification, extraction, normalization, or formatting. Reserve more capable models for tasks where their reasoning ability produces meaningful value. This can reduce cost even when the number of tokens remains approximately the same.

Avoid Repeating Large Instructions Between Agents

Multi-agent systems introduce another common source of wasted tokens. Suppose every agent receives:

10,000-token application specification
5,000-token security policy
8,000-token tool documentation
4,000-token environment description

If five agents independently receive all of it, a substantial amount of context may be duplicated. Instead, decide what each agent actually needs. For example:

Specialized agents should receive specialized context. This reduces token usage while also creating clearer boundaries between responsibilities.

A More Efficient AI Architecture

Once all of these techniques are combined, an AI application begins to look less like a direct API call and more like a conventional software architecture.

Every layer serves a purpose.

• Traditional logic prevents unnecessary AI calls.
• Retrieval reduces the amount of context.
• Application caching prevents repeated backend work.
• Prompt caching reduces repeated model processing.
• Structured responses reduce output tokens.

Together, those optimizations can make a substantial difference.

Think About Token Usage Before Every Request

When building an AI application, I find it useful to ask a few questions before sending context to the model.

• Has the model already seen this information?
• Can this stable context benefit from prompt caching?
• Can the underlying API or database result be cached?
• Does the model need the entire dataset?
• Can retrieval reduce it first?
• Does the full conversation history still matter?
• Can older history be summarized?
• Does this agent need every available tool?
• Could normal code perform this operation instead?
• Could a smaller model handle this task?
• Does the output really need to be verbose?
• Am I measuring cached and uncached token usage?

These questions often produce much larger savings than trying to shave a few sentences from a system prompt.

Caching Has Privacy and Retention Implications

Caching should not be treated purely as a performance feature. Depending on the provider and caching mechanism, cached context can have different retention characteristics. That means an architecture decision such as:

Cache this large customer dataset for an hour

may also be a:

Data retention decision
Security decision
Privacy decision
Compliance decision

Applications handling regulated, confidential, proprietary, or customer data should therefore consider caching as part of their normal data-governance architecture. Questions worth asking include:

• Where is cached content stored?
• How long does it remain?
• Can the lifetime be configured?
• Is cached content isolated appropriately?
• What happens when the cache expires?
• Can sensitive data be excluded?
• Does caching affect data-retention commitments?
• How does caching interact with compliance requirements?

Reducing token costs is valuable, but not at the expense of appropriate data handling.

Cache Freshness Matters Too

Application caching introduces another familiar software problem: stale data. Suppose an asset inventory is cached for twenty-four hours. An endpoint is compromised and removed from service ten minutes after the cache is populated. An AI security agent working against the cached inventory may still believe the endpoint is active. The architecture therefore needs to consider both:

High-value dynamic data may need short cache durations. Stable reference material can remain cached much longer. The correct balance depends on the application.

Measure Whether Caching Is Actually Working

Do not assume that adding caching automatically means the application is saving tokens. Measure it. Most major AI providers expose some form of token usage information, including cached token usage where supported. A useful telemetry record might look like:

{
  "request_id": "req-0182",
  "input_tokens": 32000,
  "cached_input_tokens": 24500,
  "uncached_input_tokens": 7500,
  "output_tokens": 1200,
  "cache_hit_percentage": 76.5
}

Over time, those values can reveal:

• prompts that rarely hit the cache;
• changing prompt prefixes;
• unnecessarily large dynamic sections;
• agents with oversized tool definitions;
• workflows repeatedly loading the same documents;
• conversations that need compaction;
• datasets that should be filtered earlier.

Caching should be observable just like database performance or application latency.

Track Tokens by Component

Looking only at total tokens can hide where the problem actually is. Instead, break the context down. For example:

Immediately, the real problem becomes clearer. The user's 250-token request is not the issue. The application architecture is. After optimization, perhaps it becomes:

Now the request is dramatically smaller, and much of the stable portion may also benefit from provider-side caching.

The Goal Is Not Simply Fewer Tokens

Reducing token usage can sound like a billing exercise. It is really an application-architecture exercise. A well-designed AI application should avoid unnecessary work at every layer:

That produces benefits beyond cost. Efficient context can improve:

• latency;
• scalability;
• context-window utilization;
• application predictability;
• agent reliability;
• infrastructure usage;
• overall user experience.

Where I Would Start

If I were optimizing an existing AI application, I would not begin by rewriting every prompt. I would first determine where the tokens are actually going. Break each request into:

• System prompt
• Tool schemas
• Conversation history
• Retrieved documents
• Tool results
• Current request
• Output

Then identify which portions are:

• Repeated
• Large
• Stable
• Retrievable
• Summarizable
• Unnecessary

That immediately tells you where the largest opportunities are. For many applications, the sequence becomes:

Efficient AI Is Mostly About Context

The biggest lesson is that token optimization is not about aggressively shortening every prompt. Sometimes a model genuinely needs a large amount of context. The problem is repeatedly sending context that adds no new information.

OpenAI, Claude, Gemini, and Grok all provide caching capabilities because repeated context has become a normal part of AI application design. But provider-side prompt caching should be only one layer. A good application should also cache its own datasets, retrieve only relevant information, summarize old state, minimize tool surfaces, choose models deliberately, and use conventional code whenever AI is unnecessary. The final architecture should move away from:

and toward:

That is the real objective.

The best AI request is not the one with the largest context window. It is the one that provides exactly the context the model needs, reuses what has already been processed, and leaves everything else outside the prompt.