Same Radeon AI PRO R9700. Same Mixture-of-Experts model. Two builds of llama.cpp — one against ROCm, one against Vulkan. The two backends do not just trade blows. They trade workloads, and the right pick depends on which phase of the request you are paying for.
If you build llama.cpp twice on the same machine — once with -DGGML_VULKAN=ON, once with -DGGML_HIP=ON — and then run identical benchmarks, you discover that the two stacks are not redundant. They are good at different things.
HIP, built on rocBLAS, throws large-matrix throughput at prefill. Vulkan, driven by the Mesa RADV driver, dispatches narrow GEMV / MoE-expert calls with lower latency than the HIP path manages today.
The takeaway for anyone shipping an agentic workload on R9700: the decision is not Vulkan-or-HIP. The decision is which one for which phase of the request.
To make this a fair comparison, both backends ran identical bits — same model file, same flash-attention flag, same prompt and decode lengths, same card, same hour.
| Variable | Value |
|---|---|
| Card | 1× AMD Radeon AI PRO R9700, gfx1201, 32 GB |
| Host | Ubuntu 24.04, ROCm 7.2.1, Mesa RADV (system Vulkan ICD) |
| Runtime | llama.cpp HEAD as of 2026-05 |
| Model | unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF, UD-Q4_K_XL, ≈17 GB |
| HIP build | -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1201 |
| Vulkan build | -DGGML_VULKAN=ON (no ROCm needed at runtime) |
| Bench | llama-bench -m <gguf> -ngl 999 -fa 1 -p 512 -n 128 |
-fa 1, decode throughput drops roughly 20% on both backends. Every number on the next slide assumes flash attention is on.A prompt of 512 tokens lands in the runtime as a single, dense matrix multiply per layer — long and thin, but fat enough that the GPU spends its time inside a real GEMM. That is what rocBLAS is tuned for, and the HIP backend in llama.cpp calls into rocBLAS for the heavy lifting.
Vulkan does not have an equivalent BLAS service. RADV is a graphics-first driver, and its compute pipeline schedules workgroups directly. For a workload that benefits from algorithmic tuning at the matmul level, that gap shows up as a 24% prefill deficit on this card.
When the request shape is "small prompt, long answer," prefill is a rounding error and this gap disappears. When the request shape is "large RAG context, short answer," prefill dominates and the HIP backend pays back its build complexity.
Decode is the opposite shape. Each token launches a stack of GEMV calls — one row of activations against a wide weight matrix — and a router pass to pick MoE experts. The math per call is small. The dispatch overhead per call is not.
That regime favors a scheduler that can fire short kernels back-to-back without queueing through a heavier abstraction layer. RADV is unusually good at that, and the R9700's RDNA4 wavefront layout maps cleanly onto its execution model. The HIP path loses 15% of its decode budget to dispatch and dequant overhead that Vulkan simply avoids.
The conclusion is not "one is better." It is "build both, decide at deploy time."
| Workload pattern | Recommended build | Why |
|---|---|---|
| Agent loop, short replies | Vulkan | Decode is the bottleneck. |
| Long-form generation | Vulkan | Decode still dominates over the run. |
| RAG with 8K+ context | HIP | Prefill cost is the gate. |
| Document summarization | HIP | Heavy ingest, short output. |
| Mixed traffic | Both | Run two instances, route at the load balancer. |
The operational cost of carrying two builds is low — both come out of the same llama.cpp source tree, both run the same GGUF — and the throughput payoff at each end of the workload spectrum is large enough to be worth the wrapper.
The two deficits on this slide are kernel-level engineering problems, not architectural ones. A small amount of work in either direction would erase the split-decision answer entirely.
Both gaps are within reach. Closing either one makes one backend dominant; closing both makes the runtime choice irrelevant.
llama.cpp's server mode is a useful demonstration, but it is not what you reach for when the request mix is real — no continuous batching, no scheduler-aware preemption, no first-class tool-call protocol. For the agentic workloads that motivated this series, the right runtimes are vLLM (which Part 2 covered) and SGLang (which has had no story on RDNA4 at all).
Part 5 takes the third runtime — SGLang — and walks through what it took to build it from source for gfx1201. There are three real blockers, and each one has a one-line fix.
setup_rocm.py gatekeeper that hard-codes CDNA targets, and the conda-env trap that quietly builds against the wrong PyTorch.