A field series

R9700 + vLLM: getting real agentic AI work out of a mainstream professional PCIe card.

Thirteen parts (so far) on bringing up an AMD Radeon AI PRO R9700 alongside a Gen 4 EPYC host, picking a runtime, and running a multi-agent workload end‑to‑end — the same card serving ~50 tok/s for one user scales to ~450–480 tok/s aggregate under 23–24 concurrent agents.

AMD Field AI Enthusiast · by Pradeep Nallimelli · June 2026

Block I · The card and the runtimes

PART 1

Meet the card — setting up LLMs on a 32 GB workstation GPU.

Why the R9700 is interesting, what 32 GB of GDDR6 actually fits, and the first working vLLM bring-up on ROCm 7.2.1.

PART 2

Mixture of Experts and the concurrency question.

Dense 14B vs. MoE 30B-A3B on the same card: why an MoE is 6.8× faster single-stream and how vLLM's batching changes the math.

PART 3

The Vulkan plot twist.

llama.cpp on Vulkan hits 108 tok/s decode on the same model that vLLM Triton does ~50 — the unexpected runtime story on RDNA4.

PART 4

HIP vs. Vulkan — the split decision.

Same model, same card, two backends: Vulkan wins decode (108 vs 92 t/s), HIP wins prefill (3,491 vs 2,822 t/s). Pick by workload.

PART 5

Bringing SGLang up on RDNA4.

Full kernel stack built end-to-end for gfx1201 (sgl_kernel, AWQ GEMV, wvSplitK INT4 MoE). What broke, what fixed it.

PART 6

SGLang bench and the cross-runtime verdict.

vLLM vs llama.cpp vs SGLang on one card, one model — the three-way comparison and why you'd pick each.

Block II · A real agentic workload, end-to-end

PART 7

What is agentic AI? End-to-end on one R9700.

The Sense·Synthesize·Verify·Decide·Act loop, why this is a multi-agent workload, and why vLLM was chosen for it.

PART 8

R9700 does synthesis, EPYC runs the agents.

Division of labor: the GPU streams tokens, the CPU does ingestion, embeddings, orchestration and pre/post-processing — constantly.

PART 9

Four days, 49,288 requests, 0 errors.

Live telemetry from a continuously-running pipeline: 35.24M prompt + 8.73M decode tokens, 368 ms TTFT, what held and what didn't.

Block III · Datacenter blueprint, one node

PART 10

Datacenter blueprint, on one node — the card isn't even sweating.

AMD's eight-stage agentic reference architecture (Gateway → Stream), mapped one-to-one onto a single R9700 + Genoa box. 79,069 requests, peak 36 concurrent, 84% KV-cache headroom.

Block IV · Where the time goes

PART 11

Where the silicon actually spends its time — profiling the loop end to end.

The same agentic pipeline across three workloads, swept 1–32 on threads vs processes. CPU-bound PDF ingestion (ProcessPool ~1.8×, GIL is the wall), network-bound web fetch (CPU idle), and LLM-bound agentic Q&A (99.8% of latency is remote generation).

Block V · The full sweep

PART 12

The full sweep — three backends, two axes, and the 8,326 revelation.

The sweep the whole series was building toward: every runtime (vLLM, HIP, Vulkan) across both halves of inference — prefill and decode — on one 30B model, one R9700. The headline is a prefill figure of 8,326 tok/s that was hiding behind the decode yardstick the entire time. Companion slide deck ↗

Block VI · The 35B only one runtime could load

PART 13

The 35B only one runtime could load — Qwen3.6-35B-A3B on a single R9700.

A hybrid linear-attention MoE lands on one 32 GB card — and only llama.cpp's converter could even open the file while vLLM-on-ROCm-navi refused the arch. The payoff: 2,752 tok/s prefill that decays only ~8% across an 8× context jump. That flat curve is the invitation for the next kernel.

Reading order: Parts 1–6 are the runtime bring-up story — read in order if you care about how the card got chosen and tuned. Parts 7–9 are the agentic workload story and stand alone if that is what you came for. Each part has a sticky sidebar to jump within sections.