prompt caching

advancedwidespread

Prompt caching means the AI provider remembers the repeating parts of requests, so it does not have to reprocess them every time, resulting in cheaper and faster responses.

In plain terms

When you send a message to a language model, much more text often goes on behind the scenes than what you typed: long instructions, attached documents, the entire conversation history. The model reads through all of this from the beginning with every question, and the provider bills for each instance.

Prompt caching eliminates this waste. It is like someone reading a 200-page document carefully once and, while it is fresh in their memory, answering questions from memory instead of paging through the whole thing again before every question. If no one asks for a long time, the memory fades and they have to read it again.

The answer will not be different or worse; the system simply saves processing the unchanged part. This matters most where the same long text repeats often: in customer service chatbots, document analysis, and AI agents working in many steps.

How it works

Everything constant (instructions, background materials, tool descriptions) goes at the beginning of the request, while variable parts, such as your fresh question, go at the end.
On the first request, the model processes the entire text, and the provider saves the processed internal state of the constant part.
On the next request, the system checks whether the beginning of the text matches a saved version letter by letter.
If it matches, the model continues from the saved state and processes only the new part; the price of text read from the cache is a fraction of the standard fee.
The saved state lasts only for a limited time: typically a few minutes or an hour, and depending on the provider and settings, even a day or longer. When it expires, it is deleted, and the process starts over.

An everyday example

You upload an 80-page contract to an AI assistant and ask ten questions about it one after another. You still have to wait a bit for the first answer, but the rest come noticeably faster because the system no longer re-reads the entire contract.

Who is affected

developerWhether caching works depends on how requests are structured, which significantly affects an AI application's cost and speed. → Code

entrepreneur, executiveThe monthly bill for a heavily used chatbot or agent can drop significantly if the system uses caching well; however, the cost of writing responses does not change.

everyonePartly thanks to this, AI assistants respond quickly even in long conversations and with large documents.

A common misconception

MythPrompt caching means the AI retrieves previously provided ready-made answers again.

In realityIt does not save the answer, but the processing of the repeated part of the input. The model formulates a new answer to every question, it just does not need to re-read what it already knows.

Limits and risks

It only works with an exact match: if even one character changes in the repeated section (for example, an inserted date), the savings are lost from the point of change onwards.
Stored state expires quickly, so it brings little benefit for infrequent requests.
It only reduces the cost and processing time of the input; writing the response costs the same and takes just as long.
With some providers, writing to the cache incurs an extra fee, so if you do not reuse the text, it may become even more expensive.
It does not work with short texts: providers only cache above a certain length; this threshold depends on the model and falls roughly between one thousand and four thousand tokens, which is a few pages of text.

Similar, but not the same

AI memory – Memory means the assistant remembers you and your past conversations long-term, while prompt caching is only a short-lived technical acceleration that 'learns' nothing about you.
response cache (semantic cache) – There, pre-written answers to similar questions are stored and reused; here, the model always generates a fresh response.
RAG (retrieval-augmented generation) – RAG decides what information is presented to the model, while prompt caching makes it cheaper to present the same information multiple times.

Related concepts

Broader

large language model, prompt, cache

Narrower

–

On the edge

token, context window, system prompt, AI-agent, API

A brief history

The technique itself (reusing the model's internal calculations) has long been known, but it only appeared in the offerings of major AI providers in 2024. That year, Google introduced it under the name 'context caching', while Anthropic and OpenAI introduced it under the name 'prompt caching'.

Where things stand

By the end of 2026, this is a basic feature of the major AI providers, working automatically in many places without separate settings. It has become one of the basic prerequisites for running agents that work in many steps over long periods economically.

Status: 2026 November

In the news

Appears in 1 Agent Age stories; the curve shows weekly occurrence in the news magazine.

Codex moves to token-based billing from April, with credits and plan limitsSep 25, 2026 · Code

Also known as

prompt caching · prompt caching · prompt caching · prompt cache · context caching · context caching · cached input · prompt-cache

Sources

This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.