next-token prediction

basicwidespread

The core operation of a language model is to estimate, based on prior text, the likelihood of each word fragment, select one, and build its response piece by piece.

In plain terms

If I say, 'He who digs a pit for others falls…', you finish without thinking: 'into it'. A language model does the same thing, only on a much larger scale. It has read vast amounts of text and learned from it what usually follows a given opening.

A token is a small building block of text: it can be a whole word, a word fragment, or a punctuation mark. The model always selects only one token at a time, adds it to the text, and asks itself again: what comes next? Long answers, poems, and program code are all created this way, piece by piece.

Your phone keyboard also suggests words, but with a small model that only looks at the immediate context. A large language model considers the entire conversation so far, which is why it can write coherent text that fits your question.

The task sounds simple, but for someone to effectively continue a medical description, a legal argument, or a calculation, they must know a great deal about the world. This is why this single practice has become such a versatile tool.

How it works

The system breaks the entered text into tokens and converts each into numbers, because the model can only work with numbers.
The model calculates the entire previous text and assigns a probability to every token in its vocabulary: how likely it is that this one comes next.
From this list, it selects one. Depending on the settings, it is not always the most probable, as there is a bit of randomness, so you may receive two different answers to the same question.
It appends the selected token to the text and repeats the step on the extended text. This continues until the model selects a special "end" token or the response reaches the length limit.
Training is the same game in reverse: the model guesses continuations in existing texts, and its internal settings are automatically adjusted in small steps based on the inaccuracy of those guesses. Chatbots are then fine-tuned through further training (for example, based on human feedback) to provide helpful answers rather than just continuing text.

An everyday example

When a chatbot's response "rolls out" word by word on the screen, you are not watching a visual effect: the model is truly generating the text then, token by token. You see the same on a small scale when your email client suggests completing your sentence in gray.

Who is affected

everyoneIf you understand that the machine writes probable continuations rather than retrieving facts from a database, you will know when to verify its response. This holds true even if the chatbot searches the web in the meantime.

parent, student, teacherThe confidently phrased but incorrect answer stems from this operation, so source verification cannot be skipped during learning. → Education & Family

developerYou measure the response length, cost, and speed in tokens, and you control its randomness with token selection settings. You always need this, whether for a simple application or a coding agent. → Code

entrepreneur, executiveAccess to AI services via APIs is typically paid based on the number of processed and generated tokens, and the usage limits of subscription packages are aligned with this. Your cost therefore depends directly on this operation.

A common misconception

MythA language model is just an enhanced autocomplete, so it cannot be capable of anything meaningful.

In realityThe basic task is indeed this, but for good guessing, the model must build a rich internal mapping of language, facts, and connections. Further training tunes it to answer questions and follow instructions. This enables it to translate, summarize, or write code, although it can also make mistakes.

Limits and risks

The model seeks the most probable continuation, not the truth: if the plausible and the correct diverge, it can state falsehoods in a convincing tone (this is called hallucination).
Due to built-in randomness, you may receive different answers to the same question, so do not draw definitive conclusions from a single response.
The model cannot delete a token once written, so an early error can persist throughout the entire response. "Thinking" models can at most correct themselves later within the text.
Precise calculations and character-by-character tasks (such as counting letters in a word) are partly more difficult for AI because it processes text in word fragments rather than individual letters and digits. Therefore, many systems call upon a calculator or program code for assistance in such cases.

Similar, but not the same

automatic text completion – The phone word predictor is based on the same principle but uses a small model and short context, while the large language model considers the entire conversation.
tokenization – Tokenization is merely splitting text into building blocks, while prediction is the step that decides which building block comes next.
search – Search finds and presents existing documents, while a language model generates new text, which is typically not a verbatim copy. Many chatbots combine both: they search and then draft based on the results.

Related concepts

Broader

language model, generative AI, machine learning, self-supervised learning

Narrower

sampling, temperature, top-p sampling

On the edge

token, tokenization, large language model, hallucination, context window

A brief history

In his 1948 work on communication theory, Claude Shannon already generated text resembling English based on the statistics of preceding letters and words. In 2003, Yoshua Bengio and colleagues used a neural network to estimate the probability of the next word. The transformer architecture introduced in 2017 enabled this process to be performed on much larger text volumes through parallelized training. The GPT models emerging from 2018 onwards were built on this.

Where things stand

At the end of 2026, the vast majority of chatbots, coding agents and text-writing tools are built on this principle. Other approaches exist, such as diffusion language models, which generate text all at once through gradual refinement. These have already appeared in some products, but the token-by-token method remains dominant in everyday use.

Status: 2026 November

Also known as

next-token prediction · next-token prediction · next-word prediction · token prediction · next-token prediction · next token prediction · next-word prediction · autoregressive text generation

Sources

This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.