GGUF

advancedwidespread

GGUF is a file format that packages a language model into a single file so you can run it on your own machine without internet.

In plain terms

GGUF is a file format, just as MP3 is for music or PDF for documents. It does not contain songs or text, but an artificial intelligence model: everything needed for the program to load it and for you to converse with it.

It is like a carefully packed moving box. Inside is the model itself (the billions of numbers where its "knowledge" resides), and written on the side of the box is what is inside, how to unpack it, and how to use it. Therefore, you usually only need to download a single file; you do not have to hunt for separate configuration files.

Most GGUF files contain compressed, in technical terms quantized, models. This shrinks the model to a fraction of its size, allowing smaller and medium models to fit on an average laptop, though they become slightly less accurate. Even compressed, powerful machines are required for the largest models. This is how the format became one of the cornerstones of home, locally run AI.

How it works

Someone takes an open model and saves it in GGUF format using a conversion program, usually in several compression levels.
The beginning of the file contains a header describing the model's structure, the rules for text segmentation (tokenization), and the conversation template. This is followed by the model's numbers, the weights.
You download the version suitable for your computer: the smaller file requires less memory but delivers lower quality, while the larger one does the opposite.
The runtime (such as llama.cpp, Ollama, or LM Studio) reads the header and automatically configures everything.
The file is structured so that the program can 'map' it directly from disk into memory, which lets the model load quickly. Depending on the runtime and the machine, the model runs on the processor, on the graphics card, or split between the two.

An everyday example

You want to try a free language model on your own laptop, so you open LM Studio and search for a model. Among the results, you see names like 'Q4_K_M.gguf, 4.9 GB'. This is a GGUF file which, after download, the program launches with a single click, and your conversation never leaves your machine.

Who is affected

developerIf you build a locally running model or a coding agent on your own machine, you will likely work with GGUF files. → Code

entrepreneur, executiveGGUF models enable processing sensitive corporate data with AI without sending it to external providers.

parent, student, teacherAfter downloading the model, you can experiment with it without a subscription or internet; smaller models can run on an average home or school computer. → Education & Family

everyoneIf you ever install an "AI running on your own machine", a GGUF file is likely working in the background.

A common misconception

MythGGUF is a specific AI model, or a smarter type of artificial intelligence.

In realityGGUF is just the packaging: the same model can exist in GGUF and other formats. Quality is determined by the packaged model and the compression level, not the format.

Limits and risks

Heavily compressed versions can deliver noticeably weaker responses than the original model, so the smallest file is not always the best choice.
Download GGUF files only from reliable sources: security vulnerabilities have already been found in the file-reading components of runner programs, and conversation templates embedded in the file can contain malicious instructions.
The format is designed for execution, not training; models are typically further trained in a different format.
The file does not change the model's license: just because it is downloadable does not mean it is freely usable for business purposes.

Similar, but not the same

Quantization – Quantization is the compression process itself, while GGUF is the file format in which the (often quantized) model is stored.
Safetensors – Safetensors stores only the model weights, with settings in separate files alongside it, whereas GGUF puts everything into a single file, optimized for local execution.
Ollama – Ollama is a runtime that downloads and launches models, while GGUF is the file format read by such programs.

Related concepts

Broader

Model Format, Helyi AI

Narrower

–

On the edge

Quantization, quantized model, llama.cpp, Ollama, Safetensors, Model Weights, Open-source model

A brief history

The GGUF format was introduced in August 2023 by the community of the llama.cpp project initiated by Georgi Gerganov, as a successor to the earlier GGML format. The goal of the new format was to be extensible and to carry all necessary model data in a single file.

Where things stand

At the end of 2026, GGUF is the most widespread distribution format for locally run language models. Popular desktop runners support it, and GGUF versions are available for most popular open language models on Hugging Face. It is not the only one: on Apple machines, for example, MLX is used, and in other environments the ONNX format is also in use.

Status: 2026 November

Also known as

GGUF · .gguf · GGUF format · GGUF file · GGUF-modell · GGUF format · GPT-Generated Unified Format · GGUF file

Sources

This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.