sampling

advancedwidespread

Sampling is the step where the language model selects one word from the possible next words based on probabilities, partly randomly.

In plain terms

The language model does not pull ready-made sentences from a drawer. At each step, it calculates how likely each word (more precisely, word piece, i.e., token) is to follow the text so far. Sampling is the moment when it actually picks one from this list.

Imagine a wheel of fortune where more probable words get larger slices and rarer ones get smaller. The model spins the wheel: it usually stops on one of the large slices, but sometimes on a small one. This is why you can get two different answers to the same question.

The wheel is 'adjustable'. If the model always chooses the largest slice, the answer is predictable but often flat and self-repetitive. If more space is given to randomness, the text becomes more varied and inventive, but the risk of nonsense or error increases.

Sampling is therefore not an error but a deliberate design decision: this is what makes machine-generated text feel natural, and why the same model can be configured for both precise and creative tasks.

How it works

The model reads the text so far and calculates a score for every token in its vocabulary about how well it fits as the next one.
It converts scores into probabilities. This is influenced by temperature: at low values, the most likely words stand out even more, while at high values, the chances are balanced out.
It usually filters the list of candidates: top-k keeps only the k most likely tokens, while top-p (nucleus) keeps just enough of the most likely ones for their combined probability to reach a given share, for example 90 percent.
From the remaining candidates, the model randomly selects one based on probability ratios.
It appends the selected token to the text, and the whole process starts over until the response is ready.

An everyday example

You ask the chatbot to write a greeting for your grandmother's 80th birthday, then click the "Regenerate" button. The second time you get a completely different text, even though the request was word for word the same. This is due to sampling: the model "drew" differently from the possible words.

Who is affected

everyoneThis is why you may receive different answers to the same question from a chatbot. It is worth asking important questions multiple times, but repetition does not replace verification.

developerBy adjusting temperature, top-p, and top-k, you decide how predictable or varied your application and the agent's behavior should be. → Code

marketerFor brainstorming and text variations, a more varied setting is useful, while for brand-consistent or legally sensitive texts, a more restrained one is better. → Marketing

lawyer, citizenSince the output can vary from run to run, an AI response is not necessarily reproducible like a traditional software result, and this matters for verifiability. → Law

A common misconception

MythIf I set the temperature to zero, the model always gives the same answer and always the correct one.

In realityAt zero temperature, the model does not sample randomly but selects the most probable token at every step. However, the most probable continuation is not necessarily true. In practice, due to technical reasons (such as minor rounding errors in calculations), small differences can still occur between two runs.

Limits and risks

Due to randomness, the same question may yield a good answer at one time and an incorrect one at another, so always verify important claims.
Higher temperature produces more varied text but makes inaccurate, fabricated, or incoherent details more common.
Overly strict settings can lead to templated, self-repeating text.
In most chat applications, you cannot see or modify these settings, and with some newer models, developers have only limited control.

Similar, but not the same

statistical sampling – In statistics, elements to be examined are selected from a large population (for example, respondents in a public opinion poll), whereas here the model selects the next word.
temperature – Temperature is an adjustable parameter of sampling, not the process itself.
hallucination – Hallucination is a confidently stated false claim by the model; sampling can increase its likelihood, but it is not the only cause.

Related concepts

Broader

text generation, decoding strategy, large language model, inference, generative AI

Narrower

temperature, top-p sampling, top-k sampling

On the edge

token, probability distribution, greedy decoding, beam search, hallucination, determinizmus, prompt

A brief history

Claude Shannon already generated English-like text in his 1948 study founding information theory by drawing letters and words based on their frequency. One of today's most widespread methods, top-p (nucleus) sampling, was described by Ari Holtzman and co-authors in 2019.

Where things stand

As of late 2026, practically every text-generating language model works with sampling, and temperature and top-p are standard settings in developer interfaces. For advanced thinking models, providers increasingly fix or narrow these values themselves.

Status: 2026 November

In the news

Appears in 1 Agent Age stories; the curve shows weekly occurrence in the news magazine.

Telex reports on AI-generated, surreal food photos in Hungarian restaurant advertisingOct 10, 2026 · Marketing

Also known as

sampling · sampling · sampling · token sampling

Sources

This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.