MoE

in English: mixture of experts

advancedwidespread

A model architecture in which most of the neural network consists of many smaller, specialized parts, and only a few of them work on each tiny piece of text (token).

In plain terms

Imagine a large outpatient clinic. Not every doctor examines every patient: at reception, staff decide whom to send you to, and only the two or three specialists you actually need deal with you. MoE does something similar inside a language model, except that it decides not per question but anew for every small piece of the text.

The traditional model activates its entire network for every word. In the MoE model, however, most of the network is divided into 'experts', and a small router part decides which few should work at any given time. This allows the model to have very large knowledge while only calculating a fraction of it for each response.

The benefit of this is speed and cost: a larger model can be run with the same amount of computing work. However, the word "expert" is misleading. These are not small models separated by topics, but internal parts of the network that specialize themselves during training, often on patterns that are difficult for humans to name.

How it works

The model breaks the text into small pieces, tokens, and processes them layer by layer.
Instead of a single large computing block, MoE layers contain several parallel experts, for example eight or even several hundred.
A small routing (gating) network assigns a score to each expert for every token and selects the best few.
Only selected experts compute; the model aggregates their results weighted by scores, while other experts rest for that token.
During training, the router and the experts learn together, and special tricks ensure that the workload is distributed evenly, so the same few experts do not always do the work.

An everyday example

You download an open model to your own machine and read in its description that it is 'MoE, 8 experts', or that only a few billion parameters are active at once out of many billions. This means the model takes up a lot of memory, yet delivers answers faster than a traditional model of the same size.

Who is affected

developerWhen selecting a model, you should know that a MoE model responds faster than a traditional model of the same size, but due to its total size, it still requires a lot of memory. → Code

entrepreneur, executiveMoE is one reason why the usage fees of large models may decrease while their capabilities increase.

everyoneYou are increasingly encountering this abbreviation in news and model descriptions; it is worth knowing that it refers to architecture, not a quality ranking.

A common misconception

MythIn a MoE model, a math expert, a legal expert, and a programming expert sit separately, and the topic of your question decides who answers.

In realityExperts are the internal components of the network that alternate token by token and layer by layer, and their specialization rarely corresponds to human topics. Many different experts participate in processing even a single sentence of yours.

Limits and risks

The computation is lower, but the memory requirement is not: all experts must be loaded, so a MoE model cannot fit on a smaller machine just because it is faster.
Its training is more delicate, because if the router distributes the load across the experts poorly, some of them stay underused and the model ends up weaker.
Total parameter count and active parameter count are two distinct metrics; when comparing models, always check which figure the manufacturer reports.
MoE alone does not make a model more reliable: it can still make mistakes and hallucinate just like any other language model.

Similar, but not the same

Multi-agent system – There, multiple independent agents converse and divide tasks among themselves, whereas MoE refers to the internal architecture of a single model.
Model Ensemble – In an ensemble, multiple full models run and their results are averaged, whereas in a MoE, only a few parts within a single model work.
Model routing – The router sends your entire question to a smaller or larger separate model, while the MoE router selects within the model, token by token.

Related concepts

Broader

Neural network, Large language model, model architecture

Narrower

Sparse Mixture of Experts, Gateway network

On the edge

Transformer, Parameter, Token, Inference, Quantization

A brief history

The idea was described in 1991 by Robert Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey Hinton. The sparse, gated variant used in today's language models was based on work published by Google researchers in 2017, building on earlier types of neural networks. The wider public got to know it through Mistral's open Mixtral model released at the end of 2023.

Where things stand

By the end of 2026, MoE is an established solution in large language models, especially in open-weight model families, where manufacturers provide both the total and active parameter counts. The internal structure of closed models is often not disclosed by manufacturers, so it is not always known whether they use it there.

Status: 2026 November

In the news

Appears in 2 Agent Age stories; the curve shows weekly occurrence in the news magazine.

Meta releases the open-weight Muse Spark Open, with 816 billion parameters, and Muse WonderOct 12, 2026 · Agent Age
JetBrains releases Mellum2.1, a 12-billion-parameter open MoE coding modelOct 8, 2026 · Code

Also known as

MoE · mixture of experts · mixture-of-experts · mixture of experts · mixture of experts · mixture of experts model · sparse MoE · MoE-modell

Sources

This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.