August 28, 2026·boracoder01

Making Qwen3.8 Flash-Next fast on RDNA4: nine refutations and a 4.6x day

A 180B MoE went from 10.4 tok/s to 102.2 single-stream (437.8 at c8) on 4x R9700 - 47.96 from an in-house engine, then a community MXFP4 checkpoint doubled it. The decomposition refuted the artisan-kernel theory: speculation 1.67x, TunableOp 3.8x, hand-written GDN kernels 0.3%. Every verdict carries a receipt.

AIRDNA4Speculative DecodingBenchmarkingMoEQuantization

Nine small sealed parcels in a three-by-three grid, each crossed out with two strokes, above a balance holding two stones against a single feather; a peacock eye-spot pattern fills the upper right corner.

Qwen3.8 Flash-Next is a 180B-parameter MoE with ~6B active per token. On a 4× AMD R9700 box (2.55 TB/s aggregate bandwidth) it decoded at 10.4 tok/s when this work started. An in-house engine took it to 47.96 with provably zero quality loss; a single M2 Ultra hit 62.3 on code. Then a community MXFP4 checkpoint and runtime doubled the AMD number to 102.2 tok/s single-stream, 437.8 aggregate at eight concurrent — and the decomposition of that final 2× refuted the most flattering theory about where it came from. Nearly every hour of the saga was spent refuting a plausible idea with a measurement, and the refutations are what made the wins findable.

The model lies about its size

"Active 6B" describes parameters, not bytes. Per decoded token, the memory bus sees ~5.4 GiB of compulsory reads at 4-bit — and only 1.1 GiB of that is the ten routed experts. The other 4.3 GiB is the always-on dense stack: 36 gated-DeltaNet linear-attention layers, 12 full-attention layers, hyper-connection projections, a shared expert, and a 636M-parameter LM head read in full every token. There is also a 51 GB n-gram embedding table (PLE) that contributes a ~2.5 KB gather per token — enormous in capacity, irrelevant in bandwidth — and an in-checkpoint MTP head for speculative decoding.

That 80/20 split decided almost every result below. Expert-parallelism parallelizes the 20%. Tensor-parallelism fights for the 80% and loses to the model's own shape. Speculation divides the whole thing by its acceptance rate. Byte math first, engineering second.

Nine refutations, each with a receipt

A rule from earlier incidents holds throughout: a negative verdict must be earned by decomposing the cost, never declared from one slow run. Every entry here has an artifact behind it.

1 · Device-resident PLE + speculation

Three escalating fit attempts on 4×32 GB. All OOM during weight load — 30.78 GiB resident per rank plus an 800 MiB load transient against 31.86 GiB of card. The table shards, the speculation head does not fit beside it. Dead by arithmetic.

2 · Piecewise graphs over a host-side gather

vLLM's compiler does not split around a CPU gather. Capture dies loudly: "Cannot copy between CPU and CUDA tensors during CUDA graph capture." The good failure mode — the silent one produces fluent nonsense.

3 · Stream-ordered waits inside HIP graphs

The upstream PLE-offload design gates a captured graph on cuStreamWaitValue32. On ROCm 7.2 the capture accepts the wait and records zero nodes — replay runs straight through and reads a stale buffer. Proven three ways: stale-value read, hipGraphGetNodes = 0, and a memset/wait/memset control recording 2 nodes. Silent corruption, not a crash.

4 · Expert-parallelism at batch 1

A measured 0.376 ms per cross-device graph split — a slope that correctly predicted a card-count delta it was not fitted to — prices EP at 192 splits/token: 9.7 tok/s projected. EP parallelizes the 20% and multiplies the bottleneck. A batch-greater-than-1 tool only.

5 · Requantizing the fat dense stack

The dense stack averages 8.5 bpw while experts get 3.5–4.9 — an obvious −32% byte lever. Built, measured: +4.3% throughput, +15% perplexity. Only a quarter of the byte savings became time, because dequant is not free: effective extraction fell from 337 to 253 GB/s on the slimmer quants. A second requant from the near-source BF16 dense shards isolated provenance and landed at +15.64% — the 15% is the bit width on this dense stack, not lossy-on-lossy stacking. The fat weights are load-bearing.

6 · Tensor-parallelism across four cards

A custom P2P allreduce came first (9.4 µs for a 5 KB payload chained 96-deep and graph-replayed — 14× under break-even; RCCL's fast small-message protocol is broken on this silicon, so rolling one was mandatory). The collective was perfect and TP4 still lost, 0.86× vs pipeline: the model has two KV heads against four cards, so attention and the LM head must mirror — only 53% of bytes shard — and ~19 µs of host ordering per graph submission adds 8.7 ms/token. The model's shape defeats the topology. (This verdict is engine-specific: the vLLM-lineage stack below runs TP4 profitably because speculation amortizes the collectives across 4-token chunks.)

7 · "Just use the fast community fork"

A benchmark post claimed 81.5 tok/s for this model on identical GPUs. Pulling the actual images and grepping their model registries showed neither tag could load the architecture at all — the number was a served alias, almost certainly a different model. Verify the artifact, not the headline. Epilogue: days later the same author published an image that can load it, and its numbers reproduced on this box to within 2% — the section below. The skepticism was about the artifact, not the author, and both verdicts were correct on their day.

8 · "The M2 Ultra's 800 GB/s will win on bandwidth"

llama.cpp on Metal: 25.5 tok/s in every placement, while the same box demonstrably sustains 434 GB/s on a dense control model. The MoE extracts 37% of the box's own demonstrated bandwidth. This architecture is per-token-overhead-bound on every backend measured — ROCm, Vulkan, Metal.

9 · Bit-identical speculative output

Spec decode diverged from serial decode at token 15–88 and the hunt treated it as a bug. It is not one: a batched DeltaNet scan cannot sum in the same order as three sequential steps, the 1-ulp difference lands in the recurrent state, and on a linear-attention model the state is the context. A V=1 control was bit-identical; the batched scan reassociates. No correct implementation can pass that gate — so the gate was wrong, not the code.

What the in-house engine earned: 10.4 to 47.96

The wins were unglamorous and additive, on a lean custom engine ported from its qwen35 backend (graph parity checked element-wise against the reference — comparing sums hid 20% element errors behind residual-stream cancellation; that method note paid for itself twice).

4x R9700, end-to-end single stream, same box, same GGUF
vLLM (eager, host PLE)                 10.4 tok/s
llama.cpp Vulkan                       25.7
engine port, 4-card pipeline           18.9   <- 58 graph splits
+ place 36 tiny gate tensors right     30.5   <- 58 -> 4 splits (+62%)
  (they are 384 bytes each)
honest end-to-end accounting           26.3   <- compute-span != token span
+ step-invariant decode graph          28.4
+ HIP graph capture (now replayable)   30.6   <- break-even at 24 tokens
+ device-side argmax                   31.7
+ MTP speculation, K=2, quality-gated  47.96  <- accept 2.6, HumanEval
                                                19/20 on BOTH arms

The single biggest jump — 18.9 to 30.5 — came from moving thirty-six 384-byte tensors onto the cards that consume them. Each one stranded on the wrong device cost two graph splits per layer. The second-biggest was speculation, which took three independent confirmations of the same acceptance profile before the number earned trust (this engine: 2.6 mean accept; an SGLang stack: 2.58; an Apple-silicon engine: 77–82% draft acceptance on code). This architecture accepts ~2.6 drafts on code and meaningfully fewer on prose, in every engine that runs it.

The Apple plot twist

Meanwhile a Zig+Metal engine (mlx-serve) shipped day-one support for this model with the n-gram table mmapped straight off the SSD — the placement the byte math had argued for from the start, since the gather is kilobytes. On an M2 Ultra it does 50.0 tok/s serial, beating a purpose-built MTP competitor's speculative arm from a 66 GB resident set. Better: its MTP was already fully wired behind a measurement gate its author had honestly left closed ("the bar for adding an arch here is a measurement"). Making the measurement settled it: code workloads pass (62.3 tok/s, 1.25×), prose regresses in every context cell, and the adaptive controller never disengages at low acceptance. So the gate stays closed, the flag goes on per-request for code traffic, and the finding went back upstream instead of a gate-flip PR.

The community checkpoint doubles it: 102.2

The endgame came off the shelf. tcclaviger published an R9700-native MXFP4/FP8 quant of the model (Qwen3.8-Flash-Next-MXFP4-FP8: 131 shards, 125,870,567,905 bytes — MXFP4 experts 64.2 GB, FP8 n-gram table 51.3 GB, FP8 attention and MTP head 2.7 GB each) plus the only runtime that can load it, the image tcclaviger/vllm:DevQwenNextFlash. On this 4× R9700 box at TP4 it decodes a single stream at 102.2 tok/s median and aggregates 437.8 tok/s at c8 — parity with the author's own TP4 table (100.3 / 440) to within 2%, on different hardware, measured before seeing his numbers for c16.

Concurrency past c8 is negative: c16 aggregates 256.3 tok/s, below c8's 437.8. The recurrent-state memory pressure of 36 DeltaNet layers inverts the usual throughput curve, and the inversion reproduced independently on both boxes. MTP acceptance runs 2.65–2.73 tokens per 4-token chunk — and greedy beats temperature 0.7 (2.73/43.2% vs 2.42/35.6%), so the acceptance gap against the author's 3.3 is checkpoint-side, not sampling.

Getting there required a provenance discovery worth stating as law. The model card's docker run example floors at ~15 tok/s — a 7× gap that is pure environment. The author's published PDF was measured against port 8078, which belongs to his docker-compose.yml, not the README: the compose adds the TunableOp environment (hipBLASLt's heuristic drops LLM-shaped GEMMs without it), the R4D_QSA kernel toggle, and different scheduler limits. Never benchmark a published stack from its README example — find the config the published numbers were actually measured on and diff the environments first.

Decomposing the 15-to-102 gap — an honest postmortem

Before running this stack, I wrote a postmortem attributing the speed gap to a missing "kernel lane" — months of hand-written HIP kernels this fleet never invested in. Ablating the stack factor by factor refuted my own postmortem:

MTP speculation (K=4)1.67× (98.0 vs 58.6 no-spec)

TunableOp environment3.8× on the no-spec path, ~0% with spec on

Hand-written HIP GDN kernels0.3% (58.8 vs 58.6 — noise)

README profile with spec onnot crippled: 98.0 vs compose 98.3

So the speed is MTP speculation, plus TunableOp-tuned GEMMs, plus the MXFP4 format whose dequant lives in-register inside the WMMA path — not artisan kernels. The kernel-lane gap is real for engines without a spec-plus-tuned-GEMM path (it is exactly why the in-house engine pays 11.7 ms/token of GDN glue), but on this stack the wins are mostly automated and architectural. The one real remaining deficit is prefill: 0.69–0.77× of the author's reference, environment-insensitive, with a two-shot all-reduce kernel measured at 42.7% of prefill GPU time — 3.0× the cost of the entire MXFP4 MoE GEMM. A profiler trace owns that next.

One more forced retraction closed the topology question: I inferred the author's prefill reference was TP2 from his PDF's device list, and recommended a two-replica TP2 production shape. The memory wall disproved it — with the 51.3 GB n-gram table offloaded to host, 69.4 GiB of weights stay VRAM-resident, and 34.7 GiB per rank does not fit a 31.86 GiB card at any utilization setting. TP2 cannot load this checkpoint on 32 GB parts; TP4 is the only feasible topology, for him and for this box.

Four-bit is not one format

Every 4-bit float family stores the same E2M1 value grid — sixteen representable numbers. The accuracy differences live almost entirely in the scale: its numeric format, and how many values share one.

MXFP4 (the OCP Microscaling format this checkpoint uses) shares one E8M0 scale — a bare power of two — across 32 values: 4.25 bits per weight, and the format the RDNA4 WMMA dequant path here is built around. NVFP4 halves the block to 16 and upgrades the scale to E4M3, buying fractional scale precision at 4.5 bpw — but its decode is native to Blackwell tensor cores, not RDNA. The lucebox ROCMFP4 family splits the difference for AMD: E2M1 values with UE4M3 scales, either dual per-16 half-block scales at ~4.5 bpw or a single per-32 "fast" variant at 4.25 bpw, imatrix-aware quantizers in-tree, kernels dispatch-gated to gfx1151/gfx12xx, and a P4MIX mode that mixes precision per-expert across the 512 experts. Refutation 5 already showed the bit width is what costs perplexity on the dense stack — so the two levers left are an in-house GPTQ/AWQ requant from the BF16 checkpoint (the acceptance gap is checkpoint-side) and a ROCMFP4 artifact that serves natively on the in-house engine.

Reproduction

Settings

Hardware4× AMD Radeon AI PRO R9700 32 GB (gfx1201), PCIe Gen5 x16 each, ~51 GB host RAM reserved for the offloaded n-gram table

Imagetcclaviger/vllm:DevQwenNextFlash @ sha256:1465c4571b6b24a0ad91ede614f91bccb938158e10a86fff7cbd56b98543e713

Checkpointtcclaviger/Qwen3.8-Flash-Next-MXFP4-FP8 — 131 safetensors shards, 125,870,567,905 bytes, verified against the HF manifest per file

Ops notesCold start 250–570 s (weight load + graph capture + TunableOp sweep) — health-check grace must be ≥600 s or the orchestrator flaps it. K must be 4 or 12: the spec chunk k+4 must divide the QSA attention block of 1648 = 2⁴ × 103, and K=12 collapses acceptance rate to 17%. No --shm-size with --ipc=host under podman.

podman run -d --network=host --ipc=host \
  --device /dev/kfd --device /dev/dri --group-add keep-groups \
  -e ROCR_VISIBLE_DEVICES=0,1,2,3 -e VLLM_PLE_CPU_OFFLOAD=1 \
  -e PYTORCH_TUNABLEOP_ENABLED=1 -e PYTORCH_TUNABLEOP_TUNING=0 \
  -e CLAV_TUNABLEOP_SWEEP=1 -e R4D_QSA=1 -e CLAV_ATTN_AUTOTUNE=1 \
  -e CLAV_GDN=0 -e VLLM_ROCM_USE_AITER=0 -e GPU_MAX_HW_QUEUES=1 \
  -e HSA_ENABLE_INTERRUPT=1 -e HSA_ENABLE_MWAITX=1 -e OMP_NUM_THREADS=8 \
  -v /path/to/Qwen3.8-Flash-Next-MXFP4-FP8:/app/models \
  -v /path/to/tunableop:/tunableop \
  docker.io/tcclaviger/vllm:DevQwenNextFlash \
  /app/models --tensor-parallel-size 4 \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 4}' \
  --kv-cache-dtype fp8 --enable-prefix-caching --enable-chunked-prefill \
  --max-num-batched-tokens 4096 --max-num-seqs 32 \
  --gpu-memory-utilization 0.95 --max-model-len 524288 \
  --compilation-config '{"cudagraph_capture_sizes": [5,10,15,20,25,30,35,40],
                         "max_cudagraph_capture_size": 40}' \
  --host 0.0.0.0 --port 8218 --served-model-name Qwen3.8-Flash-Next
# health: curl http://YOUR_HOST:8218/health  (200 when ready)

Load-bearing flags: the TunableOp env is the 3.8× no-spec factor (hipBLASLt's heuristic path drops LLM-shaped GEMMs without it); K=4 is one of only two legal spec depths; the PLE offload is what makes the weights fit at all; --max-model-len is a per-request cap, not a reservation (KV is only 12 KB/token — 12 of 48 layers carry KV at all).

Speed

single-stream (c1) decode   102.2 tok/s median (min 95.8, max 108.6)
c8 aggregate                437.8 tok/s      <- the saturation point
c16 / c24 aggregate         256.3 / 382.2    <- past c8 is NEGATIVE
TTFT p50                    267-290 ms
MTP acceptance              2.65-2.73 tok per 4-token chunk (greedy;
                            temp 0.7 is WORSE: 2.42)
prefill                     0.69-0.77x of author's reference (open)

All numbers are warm floors, never first-run pairs: an OpenAI-compatible decode bench run in 3 rounds per workload across 4 committed workloads at fixed max-tokens, repeated until consecutive rounds stop falling. The config was re-verified three separate times in one session (99.0 / 98.9 / 98.0 on code) and once more on a fresh container at session end (99.0, accept 2.71). Measured 2026-08-28.

Quality

The in-house engine's 47.96 tok/s carries the full quality gate: HumanEval N=20 run on both arms of the same binary — serial 19/20 at 31.16 tok/s vs speculative K=2 19/20 at 49.59 (1.59×), byte-identical completions on 14/20 tasks, same pass/fail verdict on 20/20. That is a same-harness baseline; zero measured quality loss. The MXFP4 production stack's speed numbers above are final, but its quality bench on that same harness is still in flight — until it lands, 102.2 is a throughput claim only, not a quality-gated one. The distinction is the whole point of the gate.

Laws that survive the day

Bench the config the numbers came from, not the README

The model card's docker-run example floors at ~15 tok/s; the compose file the author actually measured floors at ~102. Diff the environments before the first run.

Rising acceptance is a warning, not a win

MTP acceptance rose monotonically with dense-stack damage (2.08 source → 2.19 → 2.44 requants): a damaged target is easier to draft for. Never read accept as a quality signal.

Warm the page cache before dev loops

A cold 94 GB model file cost 3 minutes per iteration on a box with 251 GB of RAM. One warm pass: 10.8 GB/s reads forever after.

Throughput is bimodal per process spawn

~1 in 60 spawns lands degenerate (3.5× slow, or acceptance collapsed). One-spawn-per-arm A/Bs read the coin flip as the treatment.

Never demand bit-identity across batch shapes on recurrent models

Reassociation puts an ulp in the carried state. The correct spec gate is self-consistency plus an empirical quality A/B.

Check what the harness compiles into the scored program

A stock HumanEval stop list let the spec arm's self-written asserts into the graded code and manufactured a fake quality loss.

Effective GB/s is the only cross-stack metric

Two very different stacks both extracted ~190 GB/s before kernel fusion; the in-house engine reaches 337 after. Aggregate peak predicts nothing.

A refuted postmortem beats a flattering one

"The gap is artisan kernels" survived one day of ablation: kernels 0.3%, speculation 1.67×, tuned GEMMs 3.8×. Decompose before diagnosing.

Prior art this work leaned on: the upstream model-support and PLE-offload pull requests in vLLM (#53896, #53899), llama.cpp's qwen4exp PRs, ddalcu's mlx-serve, jundot's oMLX, tcclaviger's MXFP4-FP8 checkpoint and DevQwenNextFlash runtime, the vLLM-Radiance and libr4d kernel work by StillDeadcode, and jpezzulli's frozen SGLang trees for the RTX Pro 6000 — the most reproducible reference stacks in this space right now. Where unlicensed code was read, only interface shapes and constants were taken.

Next: the production quality bench on the MXFP4 stack, an in-house GPTQ/AWQ requant from the held BF16 checkpoint to chase the acceptance gap (2.73 vs 3.3), a ROCMFP4 lucebox artifact, and a profiler trace of the prefill all-reduce. The refutation ledger is longer than the win list — including two of my own postmortems refuted by their own follow-up measurements — and that is exactly why the win list is trustworthy.