tokenization

basicwidespread

Tokenization is the step where a program breaks text into small, numbered pieces—tokens—because the language model can only work with such units.

In plain terms

The language model does not see letters or words but numbers. Tokenization is the transformation step that cuts your entered text into pieces and assigns a number to each piece. A token can be a whole word, a word fragment, a punctuation mark, or even a single letter.

It is like LEGO: the model has a fixed set of pieces, typically tens of thousands or a few hundred thousand, and builds every text from them. Common words have ready-made pieces, while rare ones must be assembled from several smaller bricks. This lets the model write words it has never encountered before.

It is worth knowing that AI services count in tokens behind the scenes: this measures response length, how much text the model can consider at once, and the price for developer (API) access. Due to the agglutinative nature of Hungarian, words typically split into more tokens than in English, so the same text usually costs more tokens in Hungarian.

How it works

Before training, the model's creators count which letter sequences occur frequently across massive text volumes and compile a fixed vocabulary from them, typically consisting of tens of thousands or a few hundred thousand tokens.
When you type something, the tokenizer program cuts the text into dictionary elements: a common word stays intact, while a rare or long word breaks into multiple fragments.
Each token is replaced by its dictionary number, turning your sentence into a number sequence – that is what the model receives.
Based on the number sequence, the model predicts the next token, then the one after that, and so on until the response is complete.
At the end, the process reverses: the program reconstructs readable text from the issued numbers, which you see on the screen.

An everyday example

You paste a long document into a chatbot and receive a message that the text is too long, or the response cuts off mid-sentence. In such cases, you have likely hit the token limit, not a page limit: the system counts in tokens how much fits.

Who is affected

everyoneChatbots count in tokens behind the scenes how much text they can handle at once, so this determines what length of document and conversation you can work with.

entrepreneur, executiveThe fee for developer (API) access is typically calculated per token, and the usage limit of fixed-fee subscriptions also depends on this, so you can only estimate your costs if you know how many tokens a task consumes.

developerThe token count determines what fits into the model's context window, how much a call costs, and how long an agent can work without interruption. → Code

A common misconception

MythA token corresponds to a word.

In realityA token is often just a word fragment: in English, a token averages roughly three-quarters of a word, while in Hungarian, a longer, inflected word typically splits into multiple tokens. Punctuation also counts as a token, and the space usually belongs to the next word's token.

Limits and risks

Every model family has its own tokenizer – and even newer models from the same provider may count differently – so the same text can represent more or fewer tokens, making token prices impossible to compare directly.
Hungarian text generally costs more tokens than the same text in English, meaning it is more expensive and fills the quota faster.
Since the model sees tokens, not letters, it can make mistakes in counting letters, writing words backwards, or handling rhymes and wordplay; newer models have improved greatly in this area, but the error has not disappeared.
The token count can only be estimated in advance; the exact value is provided by the provider's own counter.

Similar, but not the same

asset tokenization (finance, blockchain) – There, rights attached to an asset – such as real estate or a security – are represented as a digital token; this has nothing to do with text processing.
tokenization of bank card data – A data security procedure in which the card number is replaced with a substitute code that is worthless on its own, so the merchant does not have to store and transmit the real card number.
embedding – Tokenization only splits and numbers, while embedding is the next step, assigning meaning-bearing number sequences to those numbers.

Related concepts

Broader

Natural language processing, large language model

Narrower

byte-pair encoding, subword-based tokenization

On the edge

token, context window, embedding, prompt, token-based pricing

A brief history

Splitting text into words and punctuation marks is a foundational step in computational linguistics. The byte-pair encoding now widespread in language models was described by Philip Gage in 1994 as a data compression method, then applied by Rico Sennrich and colleagues in 2015 to machine translation, breaking rare words into subword fragments.

Where things stand

As of late 2026, virtually all large language models work with tokenization, and their limits, as well as the price of developer access, are specified in tokens. Research is under way on models that work directly with bytes or characters, but the subword-based approach has remained dominant.

Status: 2026 November

In the news

Appears in 2 Agent Age stories; the curve shows weekly occurrence in the news magazine.

EBA, ESMA and EIOPA put AI and tokenisation in 2027 joint work programmeOct 12, 2026 · Money
ESMA makes AI and tokenisation an EU supervisory priority for 2027Sep 23, 2026 · Money

Also known as

tokenization · tokenization · tokenization · tokenization · tokenisation · tokenizer · tokenizer · BPE

Sources

Sennrich, Haddow, Birch: Neural Machine Translation of Rare Words with Subword Units (arXiv, 2015)

This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.