When 30 billion parameters got faster than 14 billion.

What Mixture of Experts is, in one slide.

A dense transformer reads every weight in the model for every output token. A Mixture-of-Experts transformer divides the network into many small expert sub-networks — often 128 per layer. For each token, a tiny routing network picks the best 8 experts. Only those experts run.

So even though the MoE model has 30 billion parameters sitting in VRAM, each token only touches about 3 billion of them. The other 27 billion are idle for that token, available the moment a different token wants them. If single-stream throughput is bound by "how many bytes do I read per output token," and MoE cuts that number by roughly 5×, the throughput math gets interesting fast.

Same card. Bigger model. 6.8× the speed.

The model swap moved single-stream throughput from 7.5 to 51 tokens per second. First-token latency dropped from about 410 milliseconds to 40 milliseconds — a 10× improvement. Power draw at single-user load was actually lower (216 W vs 276 W average) because compute was no longer stalled waiting on memory.

Picking the right model architecture mattered more than any kernel tuning we attempted later. This is a 6.8× speedup with no hardware change and no code change — purely a model choice, on a model that also benchmarks stronger than the 14B dense it replaced.

One to sixteen users on a single card.

The reason to put a workstation card on a developer team is concurrency. We swept from 1 to 16 simultaneous streaming chat completions with distinct prompts. Aggregate throughput rises sub-linearly — MoE pays a small bandwidth tax as concurrent users hit different experts — but per-user speed stays comfortably above reading speed all the way to 16.

The "sub-optimal kernel" warning was cosmetic. We proved it.

vLLM printed a polite warning that no tuned MoE kernel config existed for this specific GPU. Three approaches to close that warning all failed — and one caused a measurable regression, which is itself the point.