SGLang is the fastest open inference scheduler today and, until this build, a CUDA-only runtime. Getting it onto the Radeon AI PRO R9700 took three small fixes — each easy to miss, each blocking on its own — and a model swap nobody told us we needed. Here is the full bring-up walk-through, fixes and all.
vLLM gave the series its production baseline. llama.cpp gave it the surprise upset. SGLang is the third leg — and the one with the most interesting scheduler in open inference today.
SGLang ships continuous batching with prefix-cache reuse, structured-output decoding, speculative decoding, and a request-graph compiler. Production teams that already squeezed vLLM are the ones that switch to SGLang next.
None of that helps on R9700 if SGLang will not build for gfx1201. As of May 2026 the upstream wheel was CUDA-only and the source build refused the RDNA4 target outright.
| Layer | Choice |
|---|---|
| OS / driver | Ubuntu 24.04, ROCm 7.2.1 |
| Python env | conda sglang-triton36, Python 3.12 |
| PyTorch | torch 2.11.0+rocm7.2 |
| Triton | triton 3.6.0 (matches SGLang's MoE kernel registry) |
| kernels | kernels==0.10.3 — pinned, see warning below |
| SGLang source | mattbucci/2x-R9700-RDNA4-GFX1201-sglang-inference (v0.5.12 + RDNA4 patches) |
| Toolchain | GCC 13, Rust / cargo 1.96.0 via rustup (SGLang gRPC server needs Cargo) |
kernels trap. A current pip install transformers pulls kernels==0.15.1, which crashes transformers 5.6.0 at LayerRepository init with a confusing "Either a revision or a version must be specified" error. Pin kernels==0.10.3 before launching anything.Each of these surfaced as a build error or a launch crash. Each has a one-line answer. None of them are documented upstream.
sgl_kernel ships sm100 / sm120 binaries that require libnvrtc.so.13. Importing it on ROCm fails before the model even loads.site-packages/sgl_kernel/ with the source-built copy that comes out of the RDNA4 fork.setup_rocm.py hard-codes ["gfx942", "gfx950"]. A gfx1201 build is rejected by the build system itself, not the compiler."gfx1200" and "gfx1201" to the allow-list. One line.miniforge3 base env, which has no PyTorch. The first kernel that tries to #include <torch/extension.h> fails with a missing-header error that looks like a toolchain problem.--env sglang-triton36 to both scripts/build_awq_gemv.sh and scripts/build_skinny_gemms_int4.sh.$ ls ~/miniforge3/envs/sglang-triton36/lib/python3.12/site-packages/ # 1) sgl_kernel native ops — 2.9 MB, ~100 ops sgl_kernel/common_ops.cpython-312-x86_64-linux-gnu.so # includes: rotary_embedding, silu_and_mul, awq_dequantize, # moe_align_block_size, topk_softmax, gptq_gemm, # attention, grammar, kvcacheio, sampling, speculative # 2) AWQ GEMV — the narrow-shape decode kernel awq_gemv_hip_ext.so # exports: awq_gemv_hip, awq_gemv_moe_hip # 3) wvSplitK INT4 MoE GEMM — the fused MoE expert path skinny_gemms_int4_ext.so # exports: fused_moe_wvSplitK_int4_gemm
All three artifacts pass nm -D symbol checks and import without errors. The build logs land under /home/amd/pradeepn/sglang_r9700/logs/ for traceability.
Native AWQ models work. Compressed-tensors AWQ models do not, because SGLang's CT path routes the weight repack through gptq_marlin_repack — a CUDA-only kernel. On ROCm the server boots, loads weights, and then dies at process_weights_after_loading with a NameError: name 'gptq_marlin_repack' is not defined.
The fix is to pick a native AWQ build of the same model. For the A3B MoE family used through this series, that meant swapping cyankiwi/Qwen3-30B-A3B-Instruct-2507-AWQ-4bit for mattbucci/Qwen3-Coder-30B-A3B-AWQ. Same architecture, same expert count, same activation pattern — different packaging.
$ jq .quantization_config.quant_method config.json "awq" # native AWQ — routes through awq_gemv_hip, boots clean # "compressed-tensors" would route through gptq_marlin_repack — CUDA-only on ROCm
config.json before downloading. If quant_method is compressed-tensors, find a native-AWQ mirror or expect the load to fail at weight processing.$ cd /home/amd/pradeepn/sglang_r9700/repo
$ MODELS_DIR=/home/amd/AI/models TP=1 CTX=8192 MEM=0.85 \
MAX_RUNNING=8 PORT=8100 \
bash scripts/launch.sh coder-30b 2>&1 | tee logs/sglang_native.log
...
[runtime] loaded sgl_kernel ops: 103 ops registered
[runtime] gfx1201 kernel path: AWQ GEMV + wvSplitK INT4 MoE
[runtime] model: Qwen3-Coder-30B-A3B-AWQ-native
[runtime] VRAM after load: 27.7 GB / 32 GB
[runtime] The server is fired up and ready to roll
[runtime] listening on http://0.0.0.0:8100
One warning in the launch log is worth flagging now, because it sets up Part 6:
[runtime] Using default MoE kernel config. Performance might be sub-optimal.
[runtime] No tuned config for: E=128, N=384,
device_name=AMD_Radeon_AI_PRO_R9700, dtype=int4_w4a16
The hard part — getting SGLang to compile, load, and serve on a card the upstream maintainers had not targeted — is done. The next question is how fast it actually runs, and where it lands against the vLLM and llama.cpp numbers Parts 2 through 4 established.
Part 6 closes the series with the SGLang concurrency sweep, the cross-runtime synthesis chart that compares all three stacks on the same card, and the answer to a question that started this work: which runtime should an agentic workload actually pick on R9700 today.