Most teams do not need a data-center GPU. They need one good workstation card on a developer's bench. This series tests one — across vLLM, llama.cpp, dense models, Mixture-of-Experts models, and 16-way concurrency — and reports what a small AMD AI stack actually does in practice. Part one is the bring-up story, with live system snapshots from the bench.
The Radeon AI PRO R9700 is a workstation-class GPU built on AMD RDNA 4 (Navi 48, TSMC 4 nm). It targets local AI development — the workload where you want to keep model weights and data on the desk rather than in a cloud account.
| Element | Value |
|---|---|
| Architecture | AMD RDNA 4 · Navi 48 · gfx1201 |
| Compute units | 64 CUs · 4 shader engines · 2 SIMDs/CU · wavefront 32 · 4,096 stream processors |
| AI accelerators | 128 (second generation, WMMA matrix instructions) |
| Memory | 32 GB GDDR6 · 256-bit · 640 GB/s · 64 MB Infinity Cache |
| Matrix math (dense) | FP16 191 TFLOPS · INT8 383 TOPS · INT4 766 TOPS · FP8 383 TFLOPS |
| Board power | 300 W TBP · dual-slot blower · PCIe 5.0 · 4× DP 2.1a |
| List price | From $1,299 USD (board-partner cards from ASRock, ASUS, GIGABYTE, PowerColor, Sapphire, XFX) |
lspci and rocminfo.Before any model runs, you want to confirm the host actually agrees the card is there, on PCIe Gen 5 x16, with the amdgpu kernel driver bound. Two snapshots from the bench — first what the PCI bus says, then what the ROCm runtime says.
$ lspci -s 83:00.0 -vvnn
83:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:7551] (rev c0)
Subsystem: XFX Limited Device [1eae:9801]
Control: I/O+ Mem+ BusMaster+ ...
Region 0: Memory at c0000000 (64-bit, prefetchable) [size=256M]
NUMA node: 0
IOMMU group: 31
Kernel driver in use: amdgpu
Kernel modules: amdgpu
$ rocminfo | grep -A 6 gfx1201 Name: gfx1201 Uuid: GPU-b0a58d8599315bb9 Marketing Name: AMD Radeon AI PRO R9700 Vendor Name: AMD Compute Unit: 64 Shader Engines: 4 Wavefront Size: 32
1002:7551 is RDNA 4 Navi 48; the subsystem 1eae:9801 identifies this particular sample as an XFX board partner card. amdgpu driver bound, IOMMU group clean, on the right NUMA node. ROCm 7.2.1 sees it as gfx1201 — the target architecture vLLM and llama.cpp will compile for.The runtime is the open-source vLLM project, pulled from a vendor-published Docker image that pairs vLLM 0.16 with the matching ROCm compute stack. Everything in this series uses publicly available containers and publicly available models from Hugging Face. No proprietary builds.
docker run -d --name vllm-server \
--device=/dev/kfd --device=/dev/dri \
--group-add video --ipc=host --shm-size=16g --network=host \
rocm/vllm-dev:rocm7.2.1_navi_ubuntu24.04_py3.12_pytorch_2.9_vllm_0.16.0 \
vllm serve <your-AWQ-model> \
--dtype float16 --max-model-len 32768 \
--gpu-memory-utilization 0.92 \
--enable-prefix-caching \
--enable-auto-tool-choice --tool-call-parser hermes \
--served-model-name qwen3-moe --host 0.0.0.0 --port 8000
Every flag earns its place. --dtype float16 matches the activation precision the AWQ kernels expect. --max-model-len 32768 reserves attention positions for a 32K context. --gpu-memory-utilization 0.92 tells vLLM to claim 92% of VRAM for its weights + KV cache pool. --enable-prefix-caching reuses shared prompt prefixes across requests — meaningful for agent loops. --enable-auto-tool-choice --tool-call-parser hermes turns on OpenAI-compatible tool calling. A few minutes later, the OpenAI-compatible chat endpoint was live on port 8000.
The interesting lines in the vLLM startup log are not the banner — they are the memory math. With the flags above, vLLM logs exactly how it carves the card up.
(EngineCore_DP0) INFO core.py:97 Initializing a V1 LLM engine
model='cyankiwi/Qwen3-30B-A3B-Instruct-2507-AWQ-4bit'
dtype=torch.float16, max_seq_len=32768
quantization=compressed-tensors
enable_prefix_caching=True
enable_chunked_prefill=True
cudagraph_mode=FULL_AND_PIECEWISE
(EngineCore_DP0) INFO gpu_model_runner.py:4221 Model loading took 17.54 GiB memory and 15.02 seconds
(EngineCore_DP0) INFO gpu_worker.py:373 Available KV cache memory: 7.71 GiB
(EngineCore_DP0) INFO kv_cache_utils.py:1307 GPU KV cache size: 84,224 tokens
(EngineCore_DP0) INFO kv_cache_utils.py:1312 Maximum concurrency for 32,768 tokens/req: 2.57x
(EngineCore_DP0) INFO gpu_model_runner.py:5246 Graph capturing finished in 13 secs, took 0.21 GiB
(EngineCore_DP0) INFO core.py:278 init engine (profile, create kv cache, warmup) took 59.66 seconds
Once vLLM is serving, rocm-smi will say the card is roughly 92% full from the moment it boots — long before any inference request arrives. This is the most common "is something wrong?" question from someone watching the card for the first time, so it deserves a direct answer. Two snapshots from the same card, ten seconds apart.
GPU[0] Temp edge 31 °C
GPU[0] Temp junction 33 °C
GPU[0] Power 15 W
GPU[0] Engine clock 41 MHz
GPU[0] Memory clock 96 MHz
GPU[0] GPU use 3 %
GPU[0] VRAM used 27.61 GiB / 29.86 GiB
(92 % — the KV pool)
GPU[0] Temp edge 41 °C
GPU[0] Temp junction 69 °C
GPU[0] Power 339 W
GPU[0] Engine clock 3198 MHz
GPU[0] Memory clock 1258 MHz
GPU[0] GPU use 100 %
GPU[0] VRAM used 27.67 GiB / 29.86 GiB
(unchanged — KV pool reused)
--gpu-memory-utilization 0.92 tells vLLM to claim 92% of the card's VRAM at startup and hold it for the lifetime of the process — model weights plus a pre-sized KV cache pool. Reactive allocation per request would cost milliseconds of latency on every token; pre-allocation costs nothing at runtime. So the card looks "100% full" the second the engine warms up, because the cache slots are sitting there ready, not because anything is leaking. The active-request signal lives in engine clock (41 → 3,198 MHz), memory clock (96 → 1,258 MHz), power (15 → 339 W), and "GPU use" (3 → 100%) — not in VRAM-used.One user, distinct prompt, 512 output tokens, temperature 0 — the simplest possible benchmark on a dense 14B AWQ 4-bit model. Result: 7.5 tokens per second. Functional. Useful for coding-assistant work. But the card looked like it should be doing more.
The diagnostic experiment was small. We tried a model that was roughly half the parameter count at the same quantization. Throughput roughly doubled. That is the textbook signature of a memory-bandwidth-bound decode loop. At batch size 1, the GPU spends most of its time pulling weights from GDDR6, and the compute units sit waiting. Throughput is essentially memory bandwidth ÷ model bytes per token.