KV-cache

in English: KV cache (key-value cache)

professionalwidespread

The KV cache is the language model's temporary scratchpad: it keeps the partial calculations for text already processed, so the model does not have to recalculate everything for every new word.

In plain terms

The language model writes its response piece by piece, in so-called tokens (words or word fragments). Before each new piece, it considers all previous text. Without a cache, this would mean re-reading the entire conversation from the beginning for every single word.

Imagine reading a long novel where you must restart from the first page before every new sentence. Instead, you take notes: you write down what you need to know from previous pages in the margins and only read the new sentence. The KV cache is this margin note.

The benefit is speed and cost: the response is generated much faster, and the provider uses less computing capacity. The cost is memory, because notes must be stored, and the longer the text, the more space is required.

It does not change the model's knowledge or the content of the response. It delivers the same result via a shorter path.

How it works

The model breaks the entered text into tokens and calculates two number sequences for each: a "key," indicating what the token is about, and a "value," which carries its content.
The model stores these keys and values in memory at every layer. This is the KV cache.
When the model writes the next token, it only performs computation for that new token: it prepares a "query" and compares it with the keys of earlier tokens, which it retrieves from the cache. This determines how heavily it relies on the value of each earlier token.
The new token's key and value are also added to the store, so it grows with every step until the response is ready.
Providers can keep a cache of the often-repeated beginning of requests (such as a long instruction or document) between requests. This is called prompt caching.

An everyday example

You upload a fifty-page contract to a chatbot and ask ten questions about it one after another. If the provider preserves the cache between requests, you wait a bit for the first answer, but the rest start noticeably faster because the processed version of the contract is already in the store. The model still writes the answer itself token by token.

Who is affected

developerIf the beginning of your requests is always the same (system instructions, tool descriptions, documents), cache hits can cut the cost of input to a fraction and shorten wait times. Output pricing does not change. → Code

entrepreneur, executiveThe bill for long-running agents and customer service chatbots is heavily influenced by how well they utilize caching.

everyonePartly thanks to this, the chatbot continuously outputs the answer at nearly reading speed, without slowing down for every word.

A common misconception

MythThe KV cache is the model's memory, where it remembers what you previously told it.

In realityIt is only a temporary computational scratchpad for the text currently being processed, from which the model learns nothing. If deleted, the model gives the same answer, only slower and more expensively.

Limits and risks

The cache size grows with the length of the text, so very long conversations consume a lot of memory, limiting how many users a single machine can serve simultaneously.
If anything changes at the beginning of the text, even a single word, the stored portion from that point becomes invalid and must be recalculated.
At providers, the retained cache expires after a short time (typically minutes or hours), so it brings no savings for requests that repeat only rarely.
It does not improve answer quality or reduce model errors; it affects only speed and cost.

Similar, but not the same

Prompt caching – This is a priced feature offered by providers that retains the processed state of the shared beginning of requests across separate requests, whereas the KV cache is the underlying technique, which works even while a single response is being written.
Kontextusablak – The context window determines how much text the model sees at once, while the KV cache dictates how it avoids reprocessing that text.
Memory (in an AI assistant) – An assistant's memory refers to information about you and your past conversations that is permanently saved, while the KV cache is short-lived internal computational data you never see.

Related concepts

Broader

Large language model, Transzformer, Inference, Cache

Narrower

Prompt caching

On the edge

Kontextusablak, Token, Figyelmi mechanizmus, Quantization, Latency

A brief history

The technique follows from the transformer architecture introduced in 2017, whose attention mechanism works with keys and values; caching these quickly became a basic technique in text generation. In 2024, major providers began offering the version that works across requests as a separate, discounted feature.

Where things stand

By the end of 2026, it is practically a default part of running every language model, and its cross-request variant is a standard, separately priced service at the major providers. Development focuses on compressing the cache and reducing the cost of long-running agents.

Status: 2026 November

Also known as

KV-cache · KV cache · key-value cache · key-value cache · KV caching · KV caching · attention cache · attention cache

Sources

This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.