transformer
The transformer is a neural network architecture that examines every part of a text simultaneously in relation to each other; today's large language models are built on this.
In plain terms
The transformer is not a product or a program, but a blueprint: a method for how an artificial neural network is built. The models behind ChatGPT, Claude, and Gemini are all based on this blueprint. The letter T in the name GPT stands for this (Generative Pre-trained Transformer).
Imagine you need to understand a long sentence. Older systems read like someone peering through a keyhole, word by word, and by the time they reach the end, they have half-forgotten the beginning. A transformer works more like someone seeing the whole page at once, checking at each word which other word helps interpret it. This capability is called attention.
This is what made it good at following long contexts: it is much more likely to correctly work out who the pronoun at the end of a paragraph refers to at the beginning of the paragraph. Since it processes parts of the text in parallel, it can be trained much faster on huge amounts of data than its predecessors. These two properties made today's large models possible.
How it works
- It breaks the text into small pieces called tokens and converts each into a number sequence that describes its meaning.
- Since it sees everything at once, every token receives a signal about its position, otherwise word order would be lost.
- In the attention step, every token examines the others and weights how important each is for its own interpretation. In text-writing models, it can only look at the tokens preceding it, since the later ones do not yet exist.
- This step repeats across many layers. Broadly, lower layers capture grammatical relationships, while higher layers grasp more abstract meaning.
- At the end, the model calculates the probability of each possible next token, selects one (not always the most probable), and repeats the process with the extended text.
An everyday example
You type into the translator: "The dog did not cross the road because it was too tired." When translating into German, the program must decide: does "tired" refer to the dog (der Hund, so "er") or the road (die Straße, so "sie")? The transformer is likely to decide correctly because it interprets the word "tired" together with the whole sentence. Something similar happens when a chatbot correctly refers to a detail mentioned at the beginning of the conversation twenty messages later, provided it still fits within the context window.
Who is affected
A common misconception
Limits and risks
- It can only see a limited amount of text at once (this is the context window); anything beyond that is not taken into account.
- It generates probable continuations rather than verifying facts, so it can confidently write falsehoods.
- Processing longer text requires disproportionately more computation, which entails significant energy and cost demands for large models.
- Due to billions of tuned parameters, even researchers can only partially explain why a model gives a specific answer.
Similar, but not the same
- large language model – The transformer is the architecture; the large language model is the complete system built from it and trained on vast amounts of text.
- GPT – GPT is the name of OpenAI's transformer-based model family, while the transformer architecture is also used by models from many other companies.
- Neural network – The neural network is the broader umbrella term, while the transformer is one of its most widespread types today.
Related concepts
GPT, BERT, vision transformer
figyelmi mechanizmus, large language model, token, context window, embedding
A brief history
The transformer was introduced by Google researchers in 2017 in the paper "Attention Is All You Need", originally for machine translation. In 2018, BERT and the first GPT model appeared built on it, launching today's wave of large language models.
Where things stand
At the end of 2026, the transformer is the dominant architecture for language models and is also widely used in image, audio, and video processing. Some models have already introduced hybrid designs that replace some of the attention layers with another, cheaper layer type, but none of the purely alternative architectures has replaced the transformer.
In the news
- Envision deploys first commercial 800V DC power system at large AI data center in ChinaNov 2, 2026 · Ipar
- Solid-state transformers could help power the AI boom, MIT Technology Review reportsOct 26, 2026 · Ipar
- Solid-state transformers could power AI data centres as Hyosung targets the US marketSep 7, 2026 · Ipar
- US to produce more gas for AI power demand as data centres weigh on-site generationAug 31, 2026 · Ipar
- 800 VDC data centre architecture: solid-state transformers and new power supply solutionsAug 31, 2026 · Ipar
- Siemens and Reinhausen move to industrial-scale solid-state transformers for data centersAug 17, 2026 · Ipar
Also known as
transformer · transzformer · Transformer · transformer architecture · transzformer modell · transformer modell · transformer architecture · transformer model
Sources
- Vaswani et al.: Attention Is All You Need (arXiv, 2017)
- Wikipedia: Transformer (deep learning architecture)
This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.