What Is a Mixture of Experts AI Model?
A mixture of experts model uses a router to send each token to a few specialist sub-networks, so a huge model runs only part of itself at a time.
In this article
A mixture of experts, or MoE, is a type of AI model that contains many specialist sub-networks but uses only a few of them for each piece of text it processes. A small component called a router picks which ones. The result is a model with a very large total number of parameters, yet a much lower compute cost per word than a conventional model of the same size. This article explains the parts, the benefits and the costs, using Hugging Face's guide and Wikipedia's summary of the field.
Dense models versus sparse models
In an ordinary, or dense, transformer language model, every parameter takes part in processing every token. A token is a chunk of text, often a word or part of a word. If the model has 70 billion parameters, all 70 billion are used for each token.
A sparse MoE model changes this. According to Hugging Face, it replaces the feed-forward layers of a transformer with sparse MoE layers. Those layers hold most of a transformer's parameters, so changing them has a large effect. Each MoE layer contains a set of experts, often eight, and a router. For each token, only the selected experts run. This idea is called conditional computation: the work done depends on the input.
Experts and the router
The experts are usually ordinary feed-forward networks, though Hugging Face notes they could be more complex. Despite the name, they are not hand-labelled specialists such as a maths expert or a French expert. They are trained together with the rest of the model, and over training they tend to specialise in different regions of the input space, as Wikipedia describes.
The router, also called the gate, is a small learned network. It looks at each token and produces scores for the experts. A top-k choice keeps only the highest scores, with k often 1 or 2, and the token's output is a weighted combination of those experts' outputs. The router is trained at the same time as everything else by gradient descent.
A worked example: Mixtral 8x7B
Mistral AI released Mixtral 8x7B in December 2023 under the Apache 2.0 licence, according to Wikipedia. It has eight experts per layer and sends each token to two. The name suggests 56 billion parameters, but Hugging Face explains that the total is about 47 billion, because only the feed-forward layers are duplicated as experts; attention layers and others are shared. Wikipedia gives the total as 46.7 billion.
Because only two experts run per token, the compute per token is comparable to a dense model of roughly 12 billion parameters, rather than the 14 billion that two full 7-billion models would imply, since the shared layers are computed once. So the model stores about 47 billion parameters but behaves, in cost per token, like a much smaller one.
Benefits
- Faster pretraining than a dense model given the same compute budget, according to Hugging Face.
- Faster inference than a dense model with the same total number of parameters.
- Room for more total knowledge capacity without a matching rise in cost per token.
Challenges
MoE models bring their own difficulties.
- Memory. All experts must be loaded even though only some run for each token, so MoE models need a lot of video memory.
- Load balancing. A router can favour a few experts, which then train more and get chosen more, a self-reinforcing loop. Wikipedia explains that early methods added an auxiliary loss to encourage even use. A capacity factor limits how many tokens each expert may handle per batch, and overflow tokens can pass through unchanged thanks to residual connections.
- Fine-tuning. Hugging Face reports that sparse models are more prone to overfitting, although instruction tuning helps considerably.
Different developers handle balancing in different ways. Wikipedia says DeepSeek uses per-expert bias terms that shift routing toward under-used experts, avoiding an auxiliary loss, and also adds shared experts that every token uses.
A short history
The idea is older than large language models. Wikipedia traces adaptive mixtures of local experts to work in the 1990s. A 2013 paper first applied the idea to deep networks. The 2017 sparsely-gated MoE layer from Google Brain made large-scale use practical, and the Switch Transformer, which sends each token to a single expert, followed in 2022. Other named examples include Google's GLaM, with 1.2 trillion parameters and top-2 routing among 64 experts per layer, and Databricks' DBRX, released in March 2024 with 132 billion parameters and 16 experts.
Why it matters
MoE shows one way to get more out of a fixed computing budget: store a lot, use a little. The price is higher memory needs and extra engineering to keep the experts evenly used. Understanding the router, the experts and the trade-offs helps explain why a model's headline parameter count and its running cost can be so different.
Sources
Share
Keep reading
Related articles
How Attention Lets AI Models Focus on the Right Words
Attention is the mechanism that lets a language model weigh every word in a text against every other word. Here is how queries, keys and values do it, and why it gets costly.
Chain-of-Thought Prompting: Making AI Show Its Steps
Chain-of-thought prompting asks a language model to write out its reasoning steps before answering. Here is what the 2022 paper found and where the idea has limits.
What Is Temperature in AI Text Generation?
Temperature is a setting that makes an AI model's word choices more predictable or more varied. Here is the maths idea behind it and how top-k and top-p fit in.