Raising the Honeykrisp memory cap from 63 to 151 GiB, and the first 284B model on a silent Mac Studio
An M2 Ultra advertises a 97 GiB Vulkan heap but refuses allocations past 63 GiB. Two independent Mesa limits caused it — a VA layout that halves the heap and a RAM-fraction budget — and two environment variables fix it. With 151 GiB addressable, a 284-billion-parameter MoE model runs GPU-resident on a Mac Studio drawing 89 watts with its fans silent. Then the honest part: 2.2 tokens per second, decomposed. Updated 09-14: the follow-on campaign takes decode to 7.7 t/s — a quant race the cap made possible, three fused ops ported, a sign-table bug only the get-rows gate could catch, a control that turned out to be a strawman, and a live page-cache corruption caught by two digests.
The previous post on this machine ended with the decode campaign: +27% from two environment variables, and the conclusion that the remaining gap lived below the kernel source. This is the sequel. It starts with a wall nobody had measured, removes it, and ends with the biggest model this machine has ever held — running, honestly slow, and fully decomposed.
Test bench
The 63 GiB wall
The Vulkan device advertises a device-local heap of about 97 GiB. The probe
stopped at exactly 63.00 GiB — 63 one-GiB buffers allocated and bound,
the 64th refused with VK_ERROR_OUT_OF_DEVICE_MEMORY. Single buffers of 60
GiB allocated fine, so it was not a per-allocation limit. Something between
the heap and the driver cut the addressable space by a third.
Two limits, not one
Reading the Mesa source with the probe numbers in hand found two independent limits, both real, both binding at different points.
Limit one: the VA layout halves the heap. In
src/asahi/lib/agx_device.c, after computing a sparse read-only alias
divide, the code tests whether user_size == user_start and divides again.
On this device that equality holds, so the main RW heap comes out of the
second divide at 64 GiB — minus one 16 KiB guard per buffer-object VMA
(agx_va.c adds one to every BO), which is why 63 complete one-GiB
allocations fit and the guard overhead of the 64th tips it over. The probe
number and the source arithmetic agree: floor(64 GiB / (1 GiB + 16 KiB))
is 63.
Limit two: the heap budget is half of RAM. With the layout limit removed,
the same probe stopped at exactly 95.00 GiB — which is half the machine's
190 GiB. hk_physical_device.c:1162 defines
SYSMEM_HEAP_FRACTION(x) (x * 1 / 2). Mesa already ships an override —
HK_SYSMEM=<bytes> — that simply was not documented anywhere we looked.
The fix
The layout half needs a patch; the budget half needs an environment variable.
The patch adds an opt-in ASAHI_VA_MODE=wide branch to the layout function:
it keeps every existing reservation (the robustness reservation, the USC heap,
the kernel region, the guards) and decouples the sparse alias stride from the
RW heap length, yielding a 152 GiB main window. The default branch is
untouched — an unknown value or no value gives the original layout, and the
patched driver with no override measures exactly 63.00 GiB, byte-identical
behavior.
Both levers are required: HK_SYSMEM alone on the stock driver still hits the
63 GiB VA wall; wide alone hits the 95 GiB budget. The Vulkan loader has its
own trap here — it ignores ICD manifests on some mount points, so the patched
driver must be registered in /usr/share/vulkan/icd.d/ to load at all.
It costs nothing
The same 4B benchmark, same binary, same flags — only the driver changes:
And at the raw level, a compute-copy kernel and an FMA-chain kernel agree to three digits: 173.53 vs 173.46 GB/s copy, 2577 vs 2576 GFLOP/s. The patch touches address layout, not data movement, and three independent instruments confirm it.
The 284-billion-parameter moment
DeepSeek-V4-Flash Q2_K-XL is a 90.18 GiB, 284.33-billion-parameter mixture
of experts — 43 blocks, 256 experts each, 6 routed plus 1 shared per token,
MLA attention. Before today it could not load on this machine at all: the
first attempt died with BO creation failed. With the cap at 151 GiB:
Ninety gigabytes fully GPU-resident — 97 GiB of resident memory at steady state — a 284-billion-parameter model decoding on a Mac Studio drawing 75.6 watts mean, 88.9 peak, of a roughly 370-watt envelope, fans at idle speed.
The honest part: 2.2 tokens per second
That number is bad, and it decomposes cleanly. The model's own metadata — read from its GGUF header, not assumed — gives 10.46 billion active parameters per token, about 3.94 GiB of weights streamed per token. The perf-logger breakdown of one 455-millisecond token:
| component | ms | note |
|---|---|---|
| iq2_xs expert GEMMs | 136.6 | 84 calls, 8.0 GB/s effective |
| iq3_xxs expert GEMMs | 28.6 | 138–216 GFLOP/s |
| all other matmul | 45 | dense projections, router, embed |
| matmul total | 210.1 | |
| non-matmul | ~245 | attention, norms, copies, graph scheduling |
| token | 455 |
The expert kernel is the headline failure. The same GPU that streams dense Q4_0 matvec at a fitted 290 GB/s moves expert weights at 8.7 GB/s end-to-end — 33 times slower on the op that owns 65% of the matmul time. The graph's non-matmul budget is the other half, and the model architecture explains where it goes: every block carries an unfused hash-comb chain — roughly 280 source-graph operations per layer, about 12,000 per token — because the Vulkan backend has no fused implementation of that operation. CPU and CUDA have one. That is a known, bounded, upstream-able gap.
The roofline for this active-parameter set on this machine is about 74 tokens per second. Nobody is close to it on any consumer stack — but the distance to it is now three named problems, not one mystery.
A driver bug worth its own section
At 90-gigabyte scale the wide driver has a defect: every buffer-object VMA
carries a 16 KiB guard, a model allocates buffers per tensor, and both setup
and teardown go through one ioctl per VMA. Loading a 90 GB model freezes for
minutes in ASAHI_VM_BIND — userspace CPU time stops advancing entirely —
and killing the frozen process strands its exit in exit_mmap, leaving 149
GiB of kernel-held shmem with no owner and no userspace recovery short of a
reboot. The kernel logs warnings in drm_gem_shmem_get_pages_sgt under this
pattern. The 4B model is unaffected (49.27 t/s, champion speed); the bug is
strictly a VMA-count pathology. The durable fix is buffer coalescing —
fewer, larger allocations — which also reduces the per-dispatch overhead the
decode campaign fought.
The split, proven
With both machines' caps measured — 151 GiB on the Mac, 81 GiB on the Strix Halo — the same model now splits across both GPUs over the Thunderbolt fabric: a rebuilt RPC server advertises the Mac's Vulkan heap (the old build advertised CPU memory), and a client on the Strix Halo places layers on both GPUs over the verbs path, generating coherent text at 33.5 tokens per second on the 4B. Weights never cross the cable — only activations do, about 10 KiB per token against a measured 20-microsecond one-way latency. The topology that makes the big models fast is now standing.
What the hardware actually delivers
A minimal compute probe — no llama.cpp, no models, just a copy kernel and an FMA chain — settles the "is it the driver?" question:
| machine | copy (vec4) | FMA64 |
|---|---|---|
| M2 Ultra, stock driver | 178.8 GB/s | 2 577 GFLOP/s |
| M2 Ultra, wide driver | 178.8 GB/s (identical) | — |
| Strix Halo, RADV | 182.7 GB/s | 2 960 GFLOP/s |
Both machines converge within two percent on identical shaders. The raw hardware is healthy, both drivers are healthy, and the 20–33× expert-kernel gap sits entirely in the kernels, the dispatch, and the graph. That is where the next campaign goes.
Reproducing the cap fix
Power and temperature were monitored throughout: system draw, SoC rail, both fans, and every channel the SMC exposes. Nothing in this work approached a thermal or power limit. The ceilings here are structural, they are now enumerated, and each has a name.
Update, 2026-09-14: the campaign this post promised — 7.7 t/s, one real bug caught, and an honest correction
The last section named three problems: the expert kernel, the dispatch, and the unfused graph chain. A night on the graph chain moved decode from 2.20 to 7.5+ tokens per second, exposed a measurement trap worth more than the speedup, and ended with a quality question still open. In order.
The quant race the cap made possible
The 151 GiB window admits a second quantization of the same model that never fit before: Q4_K-XL with MXFP4 experts, 144.44 GiB on disk — four GiB under the raised cap. Same binary, same flags, both GPU-resident:
Decode +43% from the faster dequant despite 1.6× the bytes. The expert kernel's cost is dequant-structure-bound, not bytes-bound — nibble unpacking beats the 512-entry codebook lookup that 2-bit compression needs on a GPU with no integer dot product. Prefill is flat, as expected: prompt processing never was expert-dequant-bound. The cap raise is what made the faster arm exist at all; on the stock 63 GiB driver neither the Q4_K file nor this comparison is possible.
Three custom ops, three kernels, three gates
The "hash combiner" chain decomposes to three custom graph ops — HC_PRE,
HC_COMB, HC_POST — that the Vulkan backend had no handlers for at all.
Each became a small compute shader plus an admission gate (literal contiguous
F32 layouts only; every other shape falls back to the existing path), env-gated
behind GGML_VK_HC_COMB=1 and off by default.
The correctness gates matter more than the shaders. HC_COMB runs 20 rounds
of Sinkhorn normalization whose error compounds; it gets 23 strict parity
cases against the CPU reference — adversarial inputs (equal values, huge
opposite logits, subnormals, zero and negative scales) at three batch sizes —
with absolute-plus-relative tolerance. HC_PRE and HC_POST follow. All
green, banners confirm dispatch in the live server.
The gate caught a real bug the speed benchmark could not see. An earlier
"port CUDA's sign table" optimization of the iq2_xs dequant passed the entire
matmul suite (991/991) while being wrong: the Vulkan reference computes its
sign byte as sign7 | (popcount(sign7) << 7) — full popcount, values spanning
ten bits — while CUDA's 128-entry byte table only encodes parity. The
difference hides in one lane of sixteen (iqs % 8 == 7, second element), a
lane the matmul path never reads and the get-rows path does. The matmul gate
passed for days on a wrong table; the get-rows gate caught it in one run.
A table port is only valid where the table can represent the original
function's full range — and the proof lives in a different consumer of the
same function.
The A/B, and the correction
Same binary, same model, one environment variable apart:
+137%. Controls reproduced to the second digit across both A/B pairs. And the number is a strawman comparison, which is the part worth writing down.
The model builder has two paths for each HC stage: the fused custom op, and an unrolled views/mul/add chain of ordinary operations. An auto-probe at context creation disables the fused ops when no backend supports them. Benchmark defaults fuse them on; with the kernels off, the ops fell back to the CPU — the 3.18 control measured the model ping-ponging those three ops to the host every token. The unrolled chain, executed natively on the GPU by ordinary Vulkan ops, decodes at ~7.7 t/s (128.8 ms/token) with no kernels at all — parity with the kernel arm, not a victory over it.
The kernels matched the unrolled chain. The decode floor for this model on this machine is ~7.5–7.7 t/s either way, the 2.2 → 3.2 quant win stands on its own A/B, and the kernels' real value is the pattern: a proven admission gate, a strict-parity harness, and three less things in the graph that need the CPU. The measurement lesson is the durable part: when a control is created by a feature-flag fallback, the "improvement" may be measured against the fallback, not against the best path the system already had.
The output was garbage — six cells say it is not the kernels
Serving the model and reading what it writes (something no throughput
benchmark does) produced degenerate text: greedy decoding emits <<<<<<<
repeats and multilingual soup. The attribution matrix ran six cells — two
binaries (with and without every patch from this campaign), both model files,
mmap and direct-IO loads, GPU-only and CPU-only, fused ops on and off, the
attention indexer enabled and disabled. All six garble identically. The
kernels are exonerated: the patch-free binary garbles the same way with none
of the new code in it.
Two findings came out of the hunt anyway. First, the Mac's page cache lied, live: a page-cache read and an O_DIRECT read of the same unchanged file disagree in 39,758 bytes (disk verified clean). One five-minute command answers whether the cache is lying right now:
Load modes that bypass the cache (--load-mode dio) cost nothing and are now
everywhere on that host. Second, the remaining suspects are narrowed to the
converted model files themselves or the fork's model graph — never quality-
gated before tonight — with a perplexity probe as the next instrument. Every
speed number above is a throughput measurement; weight values do not change
dequant timing. The quality question is open, named, and next.
