Same Radeon AI PRO R9700. Same Mixture-of-Experts model. A different inference runtime. Single-stream throughput jumped from 51 to 108 tokens per second — and the card landed at about 85% of a flagship gaming GPU that costs roughly three times as much.
An open issue on the llama.cpp GitHub tracker has been collecting cross-GPU benchmarks on this exact Mixture-of-Experts model using llama.cpp's Vulkan backend — a portable graphics API that runs on basically any modern GPU regardless of vendor. The reference number for a flagship consumer gaming GPU is roughly 127 tokens per second token generation.
vLLM on the R9700 was at 51. The flagship was at 127. The R9700 has similar raw memory bandwidth and meaningfully more VRAM. So we asked the obvious question: what if the runtime is what is leaving performance on the floor, not the silicon?
llama-bench micro-benchmark with all layers offloaded to GPU and flash attention on.Two takeaways jump off the page. The same hardware went from 51 to 108 tokens per second with no silicon change — a 2.1× single-stream speedup from a runtime choice. And against the community reference, the R9700 lands at roughly 85% of a flagship gaming GPU's token-generation speed.
vLLM gets its MoE kernels from a framework that auto-generates GPU code from a high-level language. That framework has excellent, hand-tuned, vendor-supported kernels for data-center GPUs. For consumer or workstation-class GPUs from the same vendor — even though they share design DNA — the framework currently falls back to a generic, untuned code path.
llama.cpp Vulkan ships kernels hand-shaped by community contributors for the cooperative-matrix instructions that consumer GPUs expose. The kernels are mature, and the maturity gap between the two runtimes for this GPU architecture is the entire 2.1× difference above.