Mapping Honeykrisp decode: +27%, and where the other half lives
An M2 Ultra decodes a 4B model at half the rate of a machine with a third of its bandwidth. Two environment variables recover 27%, and six further experiments close every conventional explanation — the remaining 2x is provably below the kernel source and the compiler schedule.
An M2 Ultra decodes a 4B quantized model at 38.5 tokens per second through Vulkan. A Strix Halo — a third of the memory bandwidth on paper — decodes the same file at 78. Every optimization avenue on that gap was measured, one closed per experiment, and the result is a map: two environment variables recover +27%, and the rest of the gap provably does not live in the kernel source, the arithmetic, or the compiler's instruction schedule. This is the whole map, with every number.
Test bench
Two lines of upstream default, half the decode
The kernel dispatch code in ggml-vulkan.cpp tunes two things per driver:
rows per workgroup (AMD GCN and Intel get raised values) and a
shader_core_count that several memory heuristics consume. Mesa's Honeykrisp
reports VK_VENDOR_ID_MESA — matching neither vendor branch — so it receives
32-thread workgroups and a core count of zero, and every heuristic downstream
of those two values runs at its floor.
The two knobs compose multiplicatively — 1.213 × 1.038 = 1.259, measured 1.258. The direction is monotone in parallelism per row: raising rows per workgroup loses 7–34% (fewer groups), doubling the groups gains 7%, and the four-subgroup cooperative form wins outright. Strix Halo's driver makes the equivalent choice for its hardware; Honeykrisp never got the branch.
What the machine was actually doing
With the two knobs set, a perf-logger decomposition of one decode token shows the whole token in matrix-vector kernels, and the per-shape effective bandwidth explains the rest of the gap:
| shape | per dispatch | effective |
|---|---|---|
| gate+up 9728×2560 | 81.9 µs | 171 GB/s · 4864 workgroups |
| down 2560×9728 | 84.6 µs | 165 GB/s |
| q-proj 4096×2560 | 55.0 µs | 107 GB/s |
| kv 1024×2560 | 38.7 µs | 38 GB/s · 512 workgroups |
| output head q6_K | 1582 µs | 202 GB/s |
| — machine peak | 800 GB/s |
Useful FLOPs run at 0.5–2.1% of the 27 TFLOPS peak. Issued instructions — counting the full dequantization bloat, ~12.8 scalar operations per weight — sit at ~13% of issue capacity. An ALU-ablation run (same loads, checksum instead of the arithmetic) buys exactly +14.4%, matching that ceiling. Neither pipe is saturated; the kernel is latency-bound.
Fitting the six shapes gives t ≈ 34 µs + bytes/290 GB/s: every dispatch pays a fixed cost before the first byte streams. At 250 matvec dispatches per token that fixed cost is ~8.5 ms of a 20.6 ms token — 41% of every token is dispatch overhead, not work.
Closed by measurement
Each row of that ladder is an experiment, not an argument. Two deserve detail.
First: porting Apple's own Metal kernel for this exact operation — its geometry, then separately its arithmetic — loses on Honeykrisp (−10% and −17.5% respectively, 991/991 correct both times). The generic kernel's structure is already right for this machine; Apple's source is right for Apple's compiler.
Second: gate/up pair fusion. The host patch fuses all 36 layers — the perf
logger shows MUL_MAT_PAIR_GATE_UP at 36 dispatches replacing 72, and the
pair itself runs 19% faster — but end-to-end gains only +1.1% where the
fixed-cost model predicted +6%. The fixed cost was partly overlapped with
other work; removing dispatches converts to wall-clock at only ~18% of the
naive rate. Every future dispatch-count estimate gets priced at the measured
conversion, not the nominal saving.
Apple compiles the same scalar kernel
ggml's Metal kernel compiles on a Mac with xcrun -sdk macosx metal -S -O3.
The optimized IR of the reference q4_0 matvec is scalar throughout: 2-byte
nibble loads, element-by-element activation reads, one simd_sum, no vector
loads, no exotic intrinsics. The same structure our Vulkan shader uses.
And the Mesa compiler flags tell the complementary story: disabling the
backend optimizer entirely (AGX_MESA_DEBUG=noopt), or the scheduler
(nosched), moves decode by less than half a percent. The gap to Apple's ~2x
on the same silicon is not in the kernel source and not in the instruction
schedule.
What remains, and the oracle plan
Two layers sit below everything eliminated so far. Apple's final
AIR-to-AGX lowering — invisible from Linux, capturable by running macOS as a
guest under the m1n1 hypervisor and tracing what the driver uploads, then
decoding with Asahi's own disassembler. And the kernel-driver submission
pipeline, where the 34 µs fixed cost lives — open source on this machine,
profilable. A debug Mesa build (26.1.5, same commit as the release driver)
provides the other half of the comparison: this machine's own compiled machine
code for the same kernel, dumpable with AGX_MESA_DEBUG=shaders. The plan is
one line: disassemble both, diff the structure, patch what the diff names.
Reproducing the +27%
Power and thermals were monitored throughout (system draw, SoC rail, fans, every temperature channel the SMC exposes): 75.6 W mean / 88.9 W peak under full decode against a ~370 W envelope, fans unmoved. Nothing here is power- or thermally-limited. The ceiling is structural, and the map now says precisely which two structures hold it.
Update, 2026-09-14: the campaign this map called for ran — the graph chain went first, then the quant. Decode reached 7.7 t/s on this machine, one imported "optimization table" turned out wrong in a lane only the get-rows gate reads, and a control arm turned out to be measuring a CPU fallback rather than the best existing path. The full story, with the attribution matrix that cleared the kernels and caught the page cache lying again, is in the update to Raising the Honeykrisp memory cap.
