transformer
in English: transformer
The transformer is the neural network architecture that examines every part of a text simultaneously in relation to each other; almost all modern chatbots and language models are built on this.
In plain terms
The transformer is not a product or a program, but a blueprint: a way of how an artificial neural network is built. It is like the engine in vehicles: you see the car (ChatGPT, Claude, Gemini), but under the hood, almost all of them work with the same type of structure.
Earlier approaches read like someone looking at text through a keyhole: they went word by word, and by the end of a long sentence what had come at the beginning often faded. The transformer works more like an attentive reader who sees the whole page before them and can glance back, at every word, at whatever it relates to.
This capability is called "attention": the model calculates how important the other words in the text are to each word. In the sentence "Kati gave the book to Anna because she asked for it," this allows it to link the word "she" to the likely character, Anna.
Because transformers process parts of the text in parallel rather than sequentially, they can be trained on huge amounts of text. This made large language models possible, and the same architecture now works with images, audio, video, and code.
How it works
- The system breaks the text into small pieces called tokens; a token can be a whole word, a word fragment, or a punctuation mark.
- Every token becomes a number sequence describing its meaning, and the model also marks which position the token occupies, since word order matters.
- The attention mechanism calculates for every token how important the rest of the text is for its understanding, and refines its meaning based on this.
- This step is repeated in many stacked layers: broadly speaking, the lower layers capture simpler, grammatical relationships, while the upper layers capture more complex connections.
- Text-generating models assign a probability to every possible next token at the end, select one based on this, append it to the text, and then the process continues with the next token. In this way, the answer is assembled piece by piece.
An everyday example
When you ask a chatbot to summarize your lease and it correctly links the notice period on page ten with the start date on page two, the transformer's attention mechanism is at work. You see the same in translation software, when it correctly matches the verb at the end of a long sentence with the subject at the beginning.
Who is affected
A common misconception
Limits and risks
- The model can state falsehoods in a convincing tone (this is called hallucination), so always verify important data, numbers, and references.
- It can only consider a limited amount of text at once; what falls outside the context window, it does not see. The longer the text, the faster the computational cost of processing it grows.
- Training and operation require enormous computing capacity and electricity, so only a few well-funded companies can build the largest models.
- Its knowledge is current up to the time of training; it knows about recent events only if the service connects a search engine or other external source.
Similar, but not the same
- large language model – The transformer is the architecture, while a large language model is a finished system built on that architecture and trained on a huge volume of text.
- GPT – GPT is the name of one of OpenAI's model families, whose final letter refers to the transformer (Generative Pre-trained Transformer).
- transformer – A transformer is an electrical device that changes voltage; the two concepts have no connection beyond their similar names.
Related concepts
figyelmi mechanizmus, GPT, BERT
large language model, token, context window, generative AI, foundation model
A brief history
The transformer was introduced by Google researchers in 2017 in the paper "Attention Is All You Need," originally for machine translation. The attention mechanism existed earlier; the novelty was that the architecture was built almost entirely on it, abandoning word-by-word processing. In 2018, OpenAI's first GPT model and Google's BERT model were built on this, and the launch of ChatGPT at the end of 2022 introduced the technology to the general public.
Where things stand
As of late 2026, leading language models and the agents built on them are almost all based on transformers, an architecture that is also dominant in image, audio, and video processing. There are more efficient alternatives, such as state-space models (Mamba). These are increasingly used in combination with transformers in hybrid models, but neither has replaced the transformer.
In the news
- Envision deploys first commercial 800V DC power system at large AI data center in ChinaNov 2, 2026 · Ipar
- Solid-state transformers could help power the AI boom, MIT Technology Review reportsOct 26, 2026 · Ipar
- Solid-state transformers could power AI data centres as Hyosung targets the US marketSep 7, 2026 · Ipar
- US to produce more gas for AI power demand as data centres weigh on-site generationAug 31, 2026 · Ipar
- 800 VDC data centre architecture: solid-state transformers and new power supply solutionsAug 31, 2026 · Ipar
- Siemens and Reinhausen move to industrial-scale solid-state transformers for data centersAug 17, 2026 · Ipar
Also known as
transzformer · transformer · Transformer · transformer architecture · transformer architecture · transzformer modell · transformer model · transformer network
Sources
- Vaswani et al.: Attention Is All You Need (arXiv, 2017)
- Wikipedia: Transformer (deep learning architecture)
This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.