pre-training

advancedwidespread

Pretraining is the long, costly first learning phase in which an AI model acquires general language and world knowledge from vast amounts of data, primarily text in the case of language models.

In plain terms

Pre-training for a language model is like elementary and high school for a human: it does not prepare for a specific profession but provides broad general knowledge. During this phase, the model 'reads' vast amounts of text (books, articles, websites, code) and learns how language is structured and how things in the world relate to each other.

Everything else builds on this foundation. Just as someone can become a doctor, lawyer, or chef after high school, a pre-trained model can become a customer service chatbot, translator, or coding agent. This requires only much smaller, targeted training or good instructions.

This differs from the earlier approach because in the past a separate model was trained from scratch for every task, on separately collected, human-labeled data. Pre-training needs no human labelers: the text itself provides both the task and the solution.

The "P" in ChatGPT stands for exactly this: GPT is short for Generative Pre-trained Transformer.

How it works

Data collection: developers assemble a massive text corpus, then clean it and break it into small units called tokens (words, word fragments).
A guessing task: the model sees part of the text and must guess what follows. For example, after "The cat climbed the..." we expect the word "tree" from it. In other models, the task is to guess the word masked from the sentence.
Correction: the suggestion is compared with the actual continuation, and if the model erred, its internal settings are adjusted by a hair's breadth toward the correct answer. These settings are the parameters: there are billions of them, and in the largest models, even trillions.
Repetition on a massive scale: the model processes text amounting to thousands of billions (trillions) of tokens over weeks or months on thousands of specialized chips. In the process, it learns not only grammar but also facts, styles, and connections.
Building on top: the completed base model is then fine-tuned and trained for conversation and safe behavior, and only then does it become the product you use.

An everyday example

You ask a chatbot about last week's news, and it replies that its knowledge is limited to a specific date. This is the imprint of pre-training: the model's knowledge ends where the texts used for its training end. If it knows something newer, it acquires it on the fly, for example via internet search, if the service includes such a function.

Who is affected

everyonePretraining determines what the chatbot you use every day knows and does not know, including how up to date it is.

entrepreneur, executiveYou do not need to train your own model from scratch because you can adapt a ready, pre-trained model to your company, which is orders of magnitude cheaper.

developerYour work mostly begins where pre-training ends: you select a pre-trained model, then fine-tune it or steer it with instructions. → Code

lawyer, citizenMuch of the debate on copyright and data protection concerns whose texts were used for pre-training and under what permissions. → Law

A common misconception

MythA pre-trained model continuously learns from what you type during the conversation.

In realityA model's parameters do not change during a conversation. What it remembers in a conversation is derived only from the text of that conversation or a separate memory function. However, depending on settings, the provider may use conversations to train future models.

Limits and risks

The model's knowledge stops at the cutoff date of its training data; it does not know about events since then on its own.
The model also adopts the errors, prejudices, and biases of the training texts, so a confident answer is not necessarily correct.
Pretraining is extremely expensive and energy-intensive, so only a few well-funded companies can build the largest models.
There is less training text for smaller languages, including Hungarian, so models are typically weaker in these languages than in English.

Similar, but not the same

fine-tuning – Fine-tuning is the short, targeted further training of a pre-trained model for a narrower task, while pre-training provides the general foundation.
inference – Inference is the moment when the trained model answers your question; at this stage, it no longer learns but applies acquired knowledge.
retrieval-augmented generation (RAG) – RAG does not retrain the model but searches for and provides fresh documents before generating a response.

Related concepts

Broader

machine learning, deep learning, model training

Narrower

next-token prediction, masked language modeling

On the edge

self-supervised learning, foundation model, fine-tuning, large language model, training data, transfer learning

A brief history

The idea of pre-training is older: in the second half of the 2000s, deep neural networks were already pre-trained, and in the 2010s, reusing models pre-trained on the ImageNet database became standard practice in image recognition. In language models, it became a dominant method in 2018 with the launch of OpenAI's first GPT model and Google's BERT model. Both were built on the transformer architecture introduced in 2017. They demonstrated that a generally pre-trained model, with little additional training, can outperform purpose-built systems in many tasks.

Where things stand

At the end of 2026, practically every known language, image and audio model is built with pre-training, which is the industry's default method. Competition centers on the scale, cost and energy demand of training, and on the legal assessment of training data. Meanwhile, the post-pre-training stage is playing a growing role, for example reinforcement learning, as is letting the model "think" for longer while it answers.

Status: 2026 November

Also known as

pre-training · pre-training · pre-trained model · pre-training · pretraining · pre-trained model · pretrained

Sources

This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.