Follow one token through an MoE layer
In a common sparse MoE transformer, attention is followed by a routed feed-forward block. A router scores the token's current representation, selects a subset of experts, and combines their outputs using routing weights.
The Mixtral paper describes eight experts per layer with two selected for each token. Other architectures add shared experts or use different routing rules.
The interactive example shows the mechanism rather than a measured production routing trace.
- Routing decisions in this text
- 54
- Experts used per layer (of 8)
- 8 · 8 · 8 · 7 · 8 · 8
Routing depends on context and layer
The router acts on a hidden representation, not the original word in isolation. Attention has already incorporated context. The same word can therefore choose different experts in different sentences.
Each routed layer has its own parameters. Expert 3 in one layer is not a continuation of expert 3 in another. A hypothetical 500-token pass through 60 routed layers produces 30,000 token-layer selections.
That count describes routing opportunities, not a count of user-visible specialists. A legal question does not select one legal model and run it from beginning to end.
Total parameters and active parameters answer different questions
Total parameters tell you how much model data must be stored somewhere. Active parameters estimate the subset involved in a token's computation. Neither number alone determines latency or the amount of device memory required.
Keeping the full model resident avoids waiting to fetch a newly selected expert. That is a common low-latency arrangement. Offloading or distributing experts is possible, but adds transfer, scheduling, and communication costs.
As a storage-only example, 100 billion parameters at four bits occupy 50GB before metadata. If 10 billion are active for one token, the model still has 100 billion parameters to place. Activity does not shrink the stored checkpoint.
Comparing an MoE with a dense model of the same active parameter count also ignores shared layers, routing overhead, communication, and kernel efficiency. Benchmark the complete architecture.
A larger batch touches more experts
Consider E experts and k selected per token. If B tokens choose independently and each expert is equally likely, the expected number touched is E × [1 − (1 − k/E)^B]. This is an assumption-based estimate, not a routing law.
For eight experts and two choices, one token touches two. Four independent tokens touch about 5.47 experts on average; eight touch about 7.20. Correlated routing, shared experts, and load-balancing strategies change the result.
Touching more experts increases the working set. It does not make computation identical to a dense network: each token still uses its selected subset, and an expert selected by several tokens can reuse weights.
- Experts read per layer
- 57 of 256
- Share of expert weights read
- 22%
- Versus one request
- 7.2× the bytes
- Per request
- 90% of solo cost
Evaluate the placement and the workload together
Record total parameters, active parameters, quantization format, expert placement, and any offload path. Measure time to first token and decode latency at several context lengths and batch sizes.
Watch device memory, transfer traffic, and tail latency as well as aggregate throughput. A configuration that serves one user efficiently can lose that advantage when many unrelated requests touch different experts.
Do not prune supposedly irrelevant experts merely because the workload has one domain. Changing the available experts changes the model. Any pruning method needs its own quality evaluation and a reproducible artifact.
Frequently asked questions
Must every expert be in GPU memory?
No. Experts can be distributed or offloaded. Keeping them resident avoids transfer delays, while other placements need their own performance evaluation.
Does a large batch turn an MoE into a dense model?
No. A batch may touch most experts, but each token still takes a sparse route. Weight reuse and the number of operations differ from a dense model.