We finally ran the sweep the whole series was building toward — every runtime, both halves of inference, one 30‑billion‑parameter model, one R9700. The headline is a prefill figure of 8,326 tok/s. The real story is why we never saw it before: for eleven parts we were measuring in the one place the winning knob is switched off.
Every earlier part measured one runtime in isolation. Part 4 pitted HIP against Vulkan and called it a split decision. Part 6 added SGLang. But none of them swept the full grid at once and asked the blunt question a customer actually asks: for this 30B MoE on a single R9700, which stack is fastest — and fastest at what?
Three ~4‑bit paths, one base model (Qwen3-30B-A3B-Instruct-2507), flash‑attention on everywhere, everything at concurrency = 1 so the comparison is apples‑to‑apples:
The ROCm backend (rocBLAS). Build 764f1e6, weights UD-Q4_K_XL GGUF (~4.5 bpw).
RADV with KHR_coopmat matrix cores — freshly rebuilt this session. Same GGUF.
Paged‑attention server, AWQ‑4bit weights, benched live against the running :8000 endpoint.
Sweep the prompt length 512 → 2048 → 4096 and the backends stop looking alike. llama.cpp humps — it peaks near pp2048 and then sags. vLLM's paged / chunked prefill does the opposite: it keeps climbing, and by pp4096 it is nearly 1.9× HIP and 2.5× Vulkan.
| context | HIP | Vulkan | vLLM AWQ |
|---|---|---|---|
| pp512 | 3,421 | 3,078 | ~3,370 |
| pp2048 | 4,635 | 3,675 | 7,089 |
| pp4096 | 4,325 | 3,386 | 8,326 |
Flip to single‑stream token generation and the winner flips too. Decode is memory‑bandwidth‑bound, and Vulkan/RADV's KHR_coopmat path extracts more of the R9700's ~640 GB/s than rocBLAS does — 167.6 tok/s, roughly 1.8× HIP and 3.4× vLLM, clearing the old baseline‑1 number of 108 we'd carried since April.
| depth | Vulkan | HIP | vLLM |
|---|---|---|---|
| tg128 | 167.6 | 92.0 | 49.3 |
| tg256 | 153.0 | 92.8 | — |
One card, three runtimes, and the right answer is a routing rule, not a trophy.
And it scales up with context. Long‑prompt ingestion is its home turf — paged/chunked prefill beats a monolithic pass. Use it for long‑context prefill.
KHR_coopmat on RDNA4 owns single‑stream generation — 1.8× HIP. Use it for latency‑sensitive, one‑user decode.
Wins nothing outright but places second in both halves (4,635 prefill / 92 decode). The safe default when you don't know the workload's shape.
The micro‑batch sweet spot found on HIP in Part 10 reproduces on Vulkan. Same resonant hump, same location, two independent backends — which upgrades it from "a tuning trick" to a publishable claim about this 30B MoE's working set.
| backend @ pp2048 | ub = 512 | ub = 2048 | gain |
|---|---|---|---|
| llama.cpp HIP | 3,391 | 4,635 | +36.7% |
| llama.cpp Vulkan | 3,019 | 3,675 | +21.7% |
-ub cannot exceed the prompt length, so at 512 the knob is mathematically pinned and every backend looks artificially flat. Move the yardstick to pp2048 and the hump appears — no new code, +32.8% on the old "record" that supposedly "resisted all optimizations."The two GGUF paths run Q4_K_XL (~4.5 bpw); vLLM runs AWQ‑4bit. Not identical bits. But both are ~4‑bit off the same base model, and the two conclusions are each controlled the right way:
Prefill throughput is dominated by matrix compute, not bytes moved. vLLM's 2.5× lead at pp4096 comes from paged/chunked scheduling — a scheduler win, not a bpw accident.
The decode comparison that matters (Vulkan vs HIP) runs the same GGUF. Same bits, same model — so the coopmat decode win is a pure backend result.
Nothing about the silicon changed between "3,491, resists all optimization" and "8,326." Only the choice of where to measure changed — context length, backend, build freshness. Half of performance work on a new architecture is refusing to let a convenient default define the number.
So — is 8,326 a world record? Be precise, because precision is what survives a fact‑check:
The first documented, reproducible, apples‑to‑apples three‑way characterization (HIP vs Vulkan/coopmat vs vLLM AWQ) of prefill and decode for this 30B MoE on the R9700 / gfx1201 / RDNA4. Plus a genuine method finding — pp512 hides a 2.5× gap; ub=2048 is a universal resonant point — and a routing rule a customer can use tomorrow.
An absolute throughput record. Datacenter parts (MI300X, H100/H200) and even a 5090 exceed 8,326 — it's an excellent number for a 32 GB workstation card, not a global maximum. And no new kernel was invented here; the measurement discipline is the contribution.
No hand‑waving. Below is the full environment stamp and the literal commands that produced every cell in the matrix above, including the 8,326 tok/s vLLM prefill peak. Same box, same model, same flags.
764f1e6 (HIP + Vulkan) · vLLM 0.16.x navi container · model Qwen3‑30B‑A3B‑Instruct‑2507 (GGUF UD‑Q4_K_XL for llama.cpp; AWQ‑4bit for vLLM). Flash‑attention (-fa 1) is mandatory on every llama.cpp run.
# Prefill + decode, micro-batch pinned at the ub=2048 sweet spot. # -ub can never exceed prompt length, so pp512 will not show the knob. llama-bench -m Qwen3-30B-A3B-Instruct-2507-UD-Q4_K_XL.gguf \ -ngl 999 -fa 1 -b 8192 -ub 2048 \ -p 512,2048,4096 -n 128 -r 3 # -> pp2048 = 4,635 tok/s (peak HIP prefill), tg128 = 92 tok/s
# Build llama.cpp OUT OF TREE with the Vulkan backend (source is root-owned). sudo apt install -y glslc libvulkan-dev glslang-tools spirv-headers spirv-tools cmake -S llama.cpp -B /home/amd/pradeepn/vkbuild -DGGML_VULKAN=ON -DGGML_HIP=OFF cmake --build /home/amd/pradeepn/vkbuild -j # Run against the SAME GGUF so the decode win is quant-controlled. LD_LIBRARY_PATH=/home/amd/pradeepn/vkbuild/bin \ /home/amd/pradeepn/vkbuild/bin/llama-bench \ -m Qwen3-30B-A3B-Instruct-2507-UD-Q4_K_XL.gguf \ -ngl 999 -fa 1 -b 8192 -ub 2048 -p 512,2048,4096 -n 128 -r 3 # -> tg128 = 167.6 tok/s (1.8x HIP decode)
# Bring up the paged/chunked-prefill server (navi image: use 'vllm serve', not 'serve'). docker run -d --name vllm-qwen3-moe --network host --ipc host --shm-size 16g \ --device /dev/kfd --device /dev/dri/renderD128 --group-add video --group-add 109 \ --security-opt label=disable -v /home/amd/data/hf_home:/root/.cache/huggingface \ rocm/vllm-dev:rocm7.2.1_navi_ubuntu24.04_py3.12_pytorch_2.9_vllm_0.16.0 \ vllm serve cyankiwi/Qwen3-30B-A3B-Instruct-2507-AWQ-4bit \ --dtype float16 --max-model-len 32768 --gpu-memory-utilization 0.92 \ --enable-prefix-caching --served-model-name qwen3-moe --host 0.0.0.0 --port 8000 # Measure prefill at 4k context, single request (concurrency=1). python benchmarks/benchmark_serving.py --backend openai-chat \ --model qwen3-moe --host 127.0.0.1 --port 8000 \ --dataset-name random --random-input-len 4096 --random-output-len 1 \ --num-prompts 1 --max-concurrency 1 # prefill tok/s = random-input-len / TTFT -> ~8,326 tok/s at pp4096
--random-input-len (512 / 2048 / 4096) to walk the vLLM prefill curve; change -p for the llama.cpp paths. Everything else is held constant — that constancy is the whole point.Today's sweep isn't the finish line; it's the benchmark harness a purpose‑built engine proves itself against — the R9700 answer to a q36‑class CUDA prefill engine. The runway already exists in the lab notebook:
Requant experts offline and route through the wvSplitK INT4 MoE GEMM — already vindicated on 30B (75 µs, no hang, fastest INT4 path). Target prefill first.
Upcast to use RDNA4 FP8 WMMA matrix cores — the other native path to the R9700's tensor throughput.
The hybrid linear‑attention trunk (the 35B arch) needs a HIP chunked‑scan kernel — where long‑prefill kernel work pays off most.
The story then writes itself: measure first (this part), then beat it with a kernel — and every improvement lands against a number that's already honest, reproducible, and on the record.