token

basicwidespread

A token is a text fragment (word, word fragment, or punctuation mark) into which the language model splits text for processing and generation, and which is also used to measure usage.

In plain terms

The language model does not see letters or whole words, but small text fragments: these are tokens. It is like LEGO: the model assembles every sentence from pre-made blocks. Frequent words get a single block, while rarer or longer ones consist of several smaller ones.

A token is a unit of measurement, just like the kilowatt-hour for electricity. Tokens specify how much text a model can consider at once, how long a response it can generate, and providers mostly bill based on tokens.

A token is not identical to a word. In English, a token is roughly three-quarters of a word. In Hungarian, a word often splits into two, three, or even more tokens. This is mainly because the tokenization dictionary was compiled predominantly on English texts, while Hungarian is highly inflected with many suffixes. Therefore, the same meaning generally requires more tokens in Hungarian than in English.

How it works

Tokenization: when you enter a question, a program called a tokenizer cuts the text into pieces. It uses a pre-recorded "dictionary," typically containing tens or hundreds of thousands of elements.
Conversion to numbers: every token is assigned a serial number because the model actually works with numbers, not letters. Internally, the model assigns a long sequence of numbers to each serial number, which carries the token's meaning.
Prediction: based on previous tokens, the model calculates the likelihood of which token comes next. It selects one, usually from the more probable options, but not always the most likely, and appends it to the text.
Repetition: this step is repeated token by token until the answer is complete. This is why you see the text appear gradually on the screen.
Measurement: the system counts the input tokens (your question and attached materials) and the output tokens (the response). This determines the cost and decides whether everything fits within the model's context window.

An everyday example

You upload a long contract to a chatbot and receive the message that the text is too long or the conversation has reached the limit. In this case, the problem is not the page count: your text consists of more tokens than the model can handle at once.

Who is affected

everyoneThe usage limits of chatbots and the amount of text they can consider at once are measured in tokens, even if you do not see this.

entrepreneur, executiveThe bill for AI services largely depends on the number of tokens used, so understanding this unit of measurement is essential for cost planning.

developerThe token limit and token price determine how much data you can pass to the model and how much it costs to run an AI agent. → Code

parent, student, teacherKnowing that the model predicts from text fragments helps you understand why it sometimes makes mistakes in spelling, counting, or word counting. → Education & Family

A common misconception

MythOne token means exactly one word.

In realityA token can be a whole word, a word fragment, a punctuation mark, or even a single letter. In Hungarian text, a word often consists of multiple tokens.

Limits and risks

The token count varies by model because each model family tokenizes text differently: the same sentence uses fewer tokens in one and more in another.
Hungarian text typically consumes more tokens than the same content in English, so the same task may cost more and use up the quota sooner.
Not only typed text consumes tokens. Attached files and images, previous messages, and the model's response also count, and in many systems, the model's 'thinking' in the background counts too.
Long, multi-step agents consume many tokens, making their costs difficult to estimate in advance.

Similar, but not the same

word – The word is a linguistic unit, while the token is a technical segmentation unit, which can be shorter than a word and rarely longer.
kriptotoken – In the crypto world, a token refers to a digital asset recorded on a blockchain, which has nothing to do with the text fragments of language models.
access token – In IT security, a token is a digital key granting access, not a text unit.

Related concepts

Broader

large language model, generative AI, Natural language processing

Narrower

bemeneti token, kimeneti token, thinking token

On the edge

tokenization, context window, prompt, embedding, token price

A brief history

Splitting text into tokens is a long-standing procedure in computational linguistics. Modern language models largely rely on byte-pair encoding (BPE) or its variants. Philip Gage described this method in 1994 for data compression, while Rico Sennrich and colleagues applied it to machine translation in 2015.

Where things stand

By the end of 2026, the token is the general unit of measurement in the AI industry: model prices, context window sizes, and usage limits are all specified in tokens. With the spread of agents, token consumption has become one of the main AI cost items for companies.

Status: 2026 November

In the news

Appears in 35 Agent Age stories; the curve shows weekly occurrence in the news magazine.

New standard proposal would simplify how publishers opt out of AI trainingNov 4, 2026 · Marketing
WorkOS, Northflank and Trigger.dev breached in attacks involving stolen AI access tokensNov 4, 2026 · Code
Anthropic adds message threads to its API, but the token bill stays the sameNov 2, 2026 · Code
Agent shopping on Hungarian industry forums: what AI buyers need from merchants and paymentsOct 28, 2026 · Marketing
Payment and commerce infrastructure takes shape for AI agent purchasesOct 26, 2026 · Marketing
LiteLLM was poisoned using GitHub tokens leaked from Claude Code, attacker saysOct 26, 2026 · Code
Amazon tells its 'Renaissance Developer' not to chase token usageOct 26, 2026 · Code
GitHub to retire old Teams app, some Copilot models and expiring gh tokens on November 16Oct 26, 2026 · Code

Also known as

token · tokenek · tokens · text unit · word fragment · LLM-token · input token · output token

Sources

Sennrich, Haddow, Birch: Neural Machine Translation of Rare Words with Subword Units (arXiv, 2015)

This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.