Six parts in, the Radeon AI PRO R9700 has been benched on vLLM, on llama.cpp Vulkan, on llama.cpp HIP, and now on SGLang. This is the close of the series — the numbers side by side, the runtime that wins each phase, and the one tuning gap that would change the ranking.
Every runtime on this slide ran a Qwen3-30B-A3B model (same architecture, same expert count, same 3B active parameters per token). vLLM and SGLang used 4-bit AWQ packings; llama.cpp used the UD-Q4_K_XL GGUF that the unsloth team published. All four stacks ran on the same R9700 with ROCm 7.2.1 on the same host, within the same week.
| Runtime | Model packaging | Quant kernel | Notes |
|---|---|---|---|
| vLLM 0.16 | cyankiwi AWQ-4bit (CT-format) | moe_wna16 | Production baseline, Part 2 |
| llama.cpp Vulkan | unsloth UD-Q4_K_XL GGUF | RADV compute | Decode champion, Part 3 |
| llama.cpp HIP | unsloth UD-Q4_K_XL GGUF | rocBLAS | Prefill champion, Part 4 |
| SGLang v0.5.12 + RDNA4 | mattbucci Coder-30B-A3B AWQ-native | moe_wna16 (untuned) | This part |
SGLang ran with sglang.bench_serving, random-input 256 / random-output 256, concurrency sweep at N = 1, 4, 8, 16. The full bench log is teed to /home/amd/pradeepn/sglang_r9700/logs/bench_*.log.
| Concurrency | Output tok/s | Total tok/s | Mean TPOT (ms) | Mean TTFT (ms) |
|---|---|---|---|---|
| N = 1 | 44.3 | 73.8 | 62.9 | 489 |
| N = 4 | 134.6 | 284.1 | 70.0 | 427 |
| N = 8 | 234.2 | 493.9 | 85.8 | 526 |
| N = 16 | 252.4 | 580.2 | 97.3 | 4,082 |
Output throughput climbs cleanly from N=1 through N=8 — roughly linear scaling per stream — and then flattens between N=8 and N=16. The same range shows a tearaway in time-to-first-token: 489 ms at N=1, 526 ms at N=8, and 4,082 ms at N=16.
That cliff is the prefill queue saturating against an MoE kernel that has no tuned config for R9700. The server log called it out at boot — Using default MoE kernel config. Performance might be sub-optimal. Output decode survives because it dispatches narrow shapes that even an untuned kernel handles well. Prefill is where the missing tuning costs real latency.
| If you care about... | Pick | Why |
|---|---|---|
| Aggregate throughput at scale | vLLM | 381 tok/s at N=16, the highest number in the series. |
| Lowest single-stream latency | llama.cpp Vulkan | 108 tok/s decode, no scheduler overhead. |
| Highest prefill throughput | llama.cpp HIP | 3,491 tok/s pp512. |
| Best scheduler features (prefix cache, spec-dec, structured output) | SGLang | 234 tok/s at N=8, even on untuned kernels. |
There is no single winner. There is a portfolio. For an agentic deployment that values latency on the first reply and throughput across many parallel agents, the right answer today is vLLM in production with a Vulkan llama.cpp fallback for single-user interactive sessions, and an open path to SGLang once its MoE kernels are tuned for RDNA4.
SGLang's distance from vLLM is not architectural. It is a missing JSON file. The MoE kernel registry already supports R9700; it just has no entry at configs/triton_3_6_0/E=128,N=384,device_name=AMD_Radeon_AI_PRO_R9700,dtype=int4_w4a16.json. Writing that entry is a tuning sweep, not a kernel-rewrite, and a tuned config would close most of the gap to vLLM at N=8 and shorten the prefill queue that drives the N=16 TTFT cliff.
gfx1200 and gfx1201 to the setup_rocm.py allow-list. Ship a ROCm wheel.The R9700 is not a research curiosity. It is a complete, multi-runtime inference platform that ships agentic-grade LLM serving from a single workstation slot — and the gap between its current numbers and the tuned-kernel ceiling is small enough to be worth closing.