quantization

advancedwidespread

Quantization is the storage of AI model numbers in a coarser, more space-efficient format to make the model smaller and faster, at the cost of slight quality degradation.

In plain terms

A language model consists of billions of numbers called "weights," where its learned knowledge resides. Originally, each is stored with high precision and many decimal places, so a serious model occupies tens or hundreds of gigabytes and requires expensive server hardware.

Quantization is like compressing a photo into JPEG or rounding 1999.90 forint to 2000 at the store. Every number gets a rounded, shorter version. Individually, each becomes less accurate, but the overall picture changes little.

The benefit is that the model shrinks to half, a quarter, or even smaller of its size, requiring less memory and power, and often responding faster. This allows a language model to run on a laptop or phone without an internet connection.

Different from retraining or replacing the model with a smaller one: here the same model remains with the same structure, just stored more efficiently.

How it works

Starting point: the weights of a ready, trained model are usually fractional numbers stored at 16 or 32 bits, meaning they operate on a very finely graded scale.
Scale selection: the program examines the range in which numbers are distributed in each layer and divides this range into few steps, for example 16 or 256.
Rounding: every weight takes the value of the nearest step, so 8, 4, or even fewer bits suffice for storage instead of 16 or 32 bits.
Verification and refinement: the model's degradation is measured on sample texts, and sensitive parts are preserved more precisely. In more thorough methods, the model is already accustomed to rounding during training.
Execution: with smaller numbers, the machine requires less memory and usually calculates faster, so the model can run on weaker hardware.

An everyday example

You download a free language model to your laptop and see something like "Q4" or "4-bit" in the filename. This indicates you received a quantized version: the original model, perhaps 30 gigabytes, now fits into 8–9 gigabytes and runs on an average machine, though it may answer slightly less accurately.

Who is affected

developerYou must decide which quantization level fits within the available memory while keeping the quality of answers acceptable. → Code

entrepreneur, executiveQuantized models can run on their own, cheaper hardware, reducing costs and ensuring sensitive data does not leave the company.

everyoneAI functions running locally on your phone or laptop typically rely on a quantized model.

A common misconception

MythA quantized model knows the same as the original, only in a smaller space.

In realityRounding always incurs loss. With mild quantization, this is barely noticeable, but with strong compression, the model makes noticeably more errors, especially in complex reasoning, calculation, and less common languages.

Limits and risks

The stronger the compression, the greater the quality loss: at 8 bits this is usually negligible, but below 4 bits it rises rapidly.
Degradation is not uniform: the model may remain equally good at one task while visibly weakening at another, so it is worth testing on your own task.
Losses can be greater in smaller languages, including Hungarian, than what English-language measurements show.
Do not consider tests of a quantized model as results of the original model, and vice versa: manufacturer-reported scores usually refer to the full-precision version.

Similar, but not the same

model distillation – In distillation, a new, smaller model is trained to imitate a large model, whereas in quantization, the same model remains but its numbers are stored with lower precision.
pruning – Pruning completely removes less important weights from the model, while quantization keeps all weights but with lower precision.
quantum computing – Despite the similar name, quantization has nothing to do with quantum computers: it refers to rounding performed on conventional machines.

Related concepts

Broader

model compression, model optimization

Narrower

post-training quantization, quantization-aware training, 4-bit quantization

On the edge

model distillation, pruning, locally running model, open-weight model, inference

A brief history

Quantization originates from 20th-century signal processing: this is how continuous sound and images become digital data. It began to be widely applied to neural networks because of the need to run them on mobile phones; one foundational work is a 2017 paper published by Google researchers on networks that calculate using only integers.

Where things stand

By the end of 2026, quantization is standard practice: multiple quantized versions are available for almost all open-weight models, and AI features running locally on phones and laptops are built on it. Research is working on improving the quality of very aggressive compression, at around 2 bits.

Status: 2026 November

Also known as

quantization · quantization · quantization · quantization · quantisation · quantized model · PTQ (post-training quantization) · QAT (quantization-aware training)

Sources

This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.