The second baseline was supposed to be a routine "bigger model, slower numbers" run. Instead it turned into a loadability story: llama.cpp's converter recognized a brand‑new hybrid‑attention architecture and benchmarked it the same afternoon — while vLLM on the ROCm‑navi container flatly refused to register the arch at all. This is the number that exists because of the runtime that could load it.
Qwen3.6‑35B‑A3B is a multimodal, hybrid linear‑attention MoE (Qwen3_5MoeForConditionalGeneration, model_type qwen3_5_moe). Its text trunk mixes full attention with linear‑attention blocks — the same family idea behind gated‑DeltaNet. That hybrid trunk is exactly why its prefill curve behaves so differently from a plain MoE.
Hidden size 2048, head_dim 256, 16 query heads / 2 KV heads. Full‑attention interval mixed with linear‑attention blocks.
moe_intermediate 512. A far wider expert pool than the 30B's 128‑expert layout — more routing, more weight to stream.
248,320 tokens. A heavy embedding + LM head that the 30B (much smaller vocab) simply doesn't pay for.
20 GB on disk — a comfortable fit inside the R9700's 32 GB, with headroom for KV cache and activations.
Before a single token was measured, the model had to load. This is where the two paths diverged hard.
| Runtime | Could it load qwen3_5_moe? | Why |
|---|---|---|
| llama.cpp (HIP) | YES — benchmarked today | The converter registers the arch at conversion/qwen.py:625 (Qwen3_5MoeForConditionalGeneration → Qwen3_5MoeTextModel). convert_hf_to_gguf.py emits a text GGUF; HIP runs it. |
| vLLM (ROCm‑navi container) | NO — arch not registered | Container vLLM (0.16.1.dev0, built Mar 2026) knows Qwen3Next* but not qwen3_5_moe. A public AWQ export exists, but the runtime can't instantiate it. |
pip -U vllm would pull a build that drops the ROCm‑navi kernels the R9700 depends on — and no newer navi image registers the arch (navi image pushes top out at Mar 2026). So for this model, on this card, today, GGUF‑on‑HIP is the only runtime that produces a number. The converter knowing the arch is the unlock.Harness: llama-bench -m <gguf> -ngl 999 -fa 1 -p 512,2048,4096 -n 128. Same box, same flags as the Part 12 sweep — only the model changed.
| Measurement | Throughput (tok/s) | Note |
|---|---|---|
| Prefill · pp512 | 2,752.05 | Peak prefill — short context |
| Prefill · pp2048 | 2,651.03 | 4× context, −3.7% |
| Prefill · pp4096 | 2,531.20 | 8× context, −8.0% total |
| Decode · tg128 | 78.48 | Single‑stream generation |
Prefill on a standard attention stack scales badly with context — the attention matrix grows quadratically, so throughput drops sharply as prompts get longer. Qwen3.6‑35B‑A3B barely flinched: from pp512 to pp4096 it shed only ~8%. That near‑flat line is the fingerprint of the hybrid linear‑attention trunk (the ssm_a / ssm_alpha state‑space blocks), where long‑context work stays close to linear instead of quadratic.
pp512 → pp4096: 2,752 → 2,531. Almost flat where a quadratic model collapses.
Full‑attention blocks interleaved with linear/state‑space blocks keep the long‑context cost sub‑quadratic.
If context is cheap to extend, the payoff from a purpose‑built long‑prefill kernel compounds — exactly the R9700 optimization target.
Against the Part 12 30B baseline (same card, same HIP path), the 35B pays a predictable tax — bigger vocab, 256‑expert routing, head_dim 256. No surprises, and no free lunch:
| Metric | 30B-A3B (Part 12) | 35B-A3B (this run) | Delta |
|---|---|---|---|
| Prefill pp512 | ~3,491 | 2,752 | ~21% slower |
| Decode tg128 (HIP) | 92.0 | 78.5 | ~15% slower |
| Prefill decay (8× ctx) | steeper | ~8% (flat) | 35B wins |
A model whose long‑prefill cost is already sub‑quadratic is the ideal target for a hand‑written HIP kernel: the arithmetic is regular, the working set is bounded, and every microsecond saved on the linear‑attention scan is multiplied across long contexts. That is precisely the gated‑DeltaNet chunked‑scan kernel on the R9700 optimization runway — the RDNA4 answer to a q36‑class prefill engine.
Requant experts offline, route through the wvSplitK INT4 MoE GEMM — vindicated on 30B (75 µs, no hang). The 35B hang is layout‑specific, not a kernel bug.
Upcast experts to FP8 to exploit RDNA4 FP8 WMMA — a second route onto the tensor units.
A HIP scan kernel for the linear‑attention trunk — where this model's flat curve turns kernel effort into throughput.
From public weights to GGUF to numbers — the exact path.
# 1. Convert the HF checkpoint to a text GGUF (converter registers the arch). python convert_hf_to_gguf.py Qwen3.6-35B-A3B --outfile qwen36-35b-a3b-f16.gguf # 2. Quantize to Q4_K_M (~20 GB, 4.88 bpw). llama-quantize qwen36-35b-a3b-f16.gguf qwen36-35b-a3b-Q4_K_M.gguf Q4_K_M # 3. Benchmark prefill (512/2048/4096) + decode, flash-attn on. llama-bench -m qwen36-35b-a3b-Q4_K_M.gguf -ngl 999 -fa 1 -p 512,2048,4096 -n 128 # -> pp512 2752 / pp2048 2651 / pp4096 2531 / tg128 78.5 tok/s