A barrier after every dispatch
Honeykrisp, Mesa's Vulkan driver for Apple silicon, emits a full GPU barrier and cache invalidate after every single compute dispatch, whether or not anything depends on it. On a 4B model decoding on an M2 Ultra that is 547 barriers per token at 18 microseconds each, or 52% of the time. Removing the ones the application never asked for is worth 20.7%, with correctness unchanged — and a per-bit ablation then showed a single unnamed bit is the entire cost, while the tempting 2x ceiling turns out to compute garbage.
A Mac Studio on my desk runs Linux and decodes a 4B model on its GPU through Mesa's Vulkan driver. For weeks the interesting question was the shader compiler, and three real defects came out of it. This post is about the one that turned out to be larger than all three together, and it is not in the compiler at all.
Every compute dispatch Honeykrisp records is followed by a full AGX barrier and cache invalidate. Not the ones that need ordering — all of them. On this model that is 547 dispatches per token and 547 barriers, at about 18 microseconds each, which is 52% of the time it takes to produce a token.
The driver says so itself, in a comment I will quote in full later, and the reason is honest: nobody knows what the bits in that packet mean, so all of them are set, after every launch, to be safe.
Test bench
One machine. No PCIe link table, because the GPU is on the SoC.
Every arm below runs on that same binary with an environment variable moved. The variable changes one branch in the driver; nothing else differs.
Where a token actually goes
The honest starting point is not a ratio against another vendor's box. It is the fraction of this machine's own 800 GB/s that the model manages to use. At the start of this post's work that was 122 GB/s, or 15.4%.
So I decomposed a token. Each matrix-vector shape in the model, measured on its own:
| shape | per layer | µs | MB | GB/s | % of 800 |
|---|---|---|---|---|---|
| 9728 x 2560 (gate, up) | 2 | 70.30 | 14.01 | 199.3 | 24.9% |
| 2560 x 9728 (down) | 1 | 74.28 | 14.01 | 188.6 | 23.6% |
| 4096 x 2560 (q) | 1 | 44.26 | 5.90 | 133.3 | 16.7% |
| 2560 x 4096 (o) | 1 | 45.10 | 5.90 | 130.8 | 16.3% |
| 1024 x 2560 (k, v) | 2 | 26.97 | 1.47 | 54.7 | 6.8% |
Two things fall out of that table. The first is that bandwidth falls as the operation gets smaller, which is the signature of a fixed cost per operation rather than a bandwidth limit. Fitting time against bytes gives about 21.5 microseconds of fixed cost and 282 GB/s at the margin.
The second is arithmetic I had been getting wrong for days. Those shapes sum to
2049.9 MB, and the file on disk is 2369.8 MB. The missing 320 MB is
token_embd, which this model ties to its output head — a q6_K matvec of
151936 x 2560 that reads 319 MB every single token, 13.5% of all the bytes,
and which my decomposition had simply omitted. Measured on its own it runs at
369 GB/s, 46% of the part's peak, which is the fastest thing in the model
and proof that neither the hardware nor the kernel has a low ceiling.
Then I counted dispatches. The driver already prints this under
ASAHI_MESA_DEBUG=perf — which, incidentally, is a different variable from the
AGX_MESA_DEBUG that controls the compiler, a distinction that cost me a
confusing empty log:
| quantity | per token |
|---|---|
| queue submissions | 11 |
| compute dispatches | 547 |
| CDM barriers | 547 |
Exactly one barrier per dispatch. Put that beside the fit and the token comes out in full: 547 x 21.5 µs of fixed cost is 11.76 ms, the weights at 282 GB/s are 7.27 ms, and the sum is 19.03 ms against 19.34 ms measured. Two numbers taken independently, landing within 2%.
The flag that looks like the experiment
Honeykrisp ships a debug flag called nobarrier. It is exactly the name you
would want, and I ran it first:
| arm | tok/s |
|---|---|
| stock | 51.8 |
| HK_PERFTEST=nobarrier | 52.2 |
Nothing. If I had stopped there I would have written off barriers entirely, and the write-off would have been wrong.
HK_PERFTEST=nobarrier skips hk_CmdPipelineBarrier2, which ends the compute
control stream. But hk_EndCommandBuffer runs merge_control_streams(), whose
whole job is to stitch adjacent compute streams back together — the driver's
own comment calls the split "sloppiness" it is cheap to undo later. So the flag
removes a split that was going to be merged anyway. It never touched the thing
that actually serializes the GPU.
The thing that does serialize it
The serialization is one line further down, in the dispatch path itself
(src/asahi/vulkan/hk_cmd_dispatch.c):
hk_cdm_cache_flush emits an AGX CDM_BARRIER packet. It runs after every
launch, unconditionally, with no reference to whether anything downstream reads
what that launch wrote. And the packet it emits is this
(src/asahi/libagx/libagx_dgc.h), with the driver's own explanation:
That comment is the finding. This is reverse-engineered hardware with no documentation, the conservative choice is the correct one to ship, and the author wrote down both the reason and the invitation to revisit. The motivating case was blits — pixel-backend and texture caches, a graphics path that has nothing to do with a chain of matrix-vector products.
So: too often, and too wide. Both are measurable.
How much is it worth at most
The driver has no switch for this, so I added one — an environment variable
that makes hk_cdm_cache_flush return immediately, default off, stock path
byte-identical. The results are wrong with it set. That is fine; the question
is only what the barrier costs.
| arm | tok/s | correct |
|---|---|---|
| stock | 52.4 | yes |
| barrier skipped entirely | 109.0 | no |
2.08x. Spread over 547 barriers, one barrier costs 18.1 microseconds. For scale, the smallest matvec in the model takes 26.7 µs to do its entire job.
Two halves, measured separately
Too often
The barrier belongs where the application asks for ordering, not after every launch. llama.cpp's Vulkan backend already tracks which operations alias — it compares byte ranges within a buffer and only synchronizes on a real overlap — so the number it asks for is the number genuinely needed:
| quantity | per token |
|---|---|
| compute dispatches | 547 |
| barriers Honeykrisp emitted | 547 |
| barriers llama.cpp asked for | 366 |
The change is to emit the barrier where the Vulkan API asks for it. Application dispatches drop the unconditional flush; driver-internal dispatches — draws, meta operations, queries, the blit paths the comment was about — keep today's behaviour exactly.
The placement matters more than it first appears. Every synchronization point
in the driver funnels through hk_cmd_buffer_end_compute(): pipeline barriers,
vkCmdWaitEvents2, vkCmdSetEvent2, the query paths, and end of command
buffer. My first version emitted the barrier at the two sites I had been
looking at, which silently left an application that orders its compute with
events instead of barriers with no ordering at all. Putting it in
end_compute itself covers all eight by construction, and it has to go in
before the stream is ended, because merge_control_streams() may rejoin that
stream and only a barrier inside it survives the merge.
Measured effect, which is the thing to check rather than the flag:
| arm | dispatches/token | barriers/token | tok/s |
|---|---|---|---|
| stock | 547 | 547 | 52.5 |
| barrier where Vulkan asks | 547 | 370 | 63.4 |
Too wide
The other half is the packet. I made the bit pattern settable and swept it,
treating bit N of the mask as bit N of the packet, so the stock pattern is
0x000fffff. Speed first:
| mask | tok/s |
|---|---|
| 0x000fffff (stock) | 52.4 |
| 0x0000ffff | 52.7 |
| 0x000000ff | 56.1 |
| 0x0000007f | 99.3 |
| 0x0000000f | 100.0 |
| 0x00000000 (empty packet) | 108.9 |
One bit does almost all of it. Going from 0x7f to 0xff — setting bit 7
alone — costs 99.3 tok/s down to 56.1, a factor of 1.77.
Speed is not the gate, though, and this is where the sweep earns its keep:
| mask | tok/s | test-backend-ops |
|---|---|---|
| 0x000fffff (stock) | 52.4 | 15937/16613 |
| 0x000000ff | 56.1 | 15937/16613 |
| 0x000000f7 | 56.0 | 15937/16613 |
| 0x0000007f | 99.3 | segfault |
| 0x0000003f | 99.3 | segfault |
| 0x0000002f | 100.3 | segfault |
| 0x0000001f | - | segfault |
| 0x0000000f | 100.0 | segfault |
| 0x00000000 | 108.9 | segfault |
Bit 7 is load-bearing. Every mask that clears it crashes the process rather
than merely computing wrong values, which says it protects something
structural. Bits 8 through 19 are pure cost — dropping them is worth 7.0% with
the suite unchanged. Bit 3, usc_cache_inval, turns out not to be needed for
this workload at all (0xf7 passes), and is nearly free anyway.
What shipped
| model | stock | fixed | delta |
|---|---|---|---|
| Qwen3-4B Q4_0 | 52.41 | 63.26 | +20.7% |
| Qwen3-4B Q4_K_M | 49.86 | 58.40 | +17.1% |
Narrowing the bit pattern on top of that is worth a further 1.6 tok/s — 64.8 on Q4_0 — but it is measured on a build carrying the sweep knob, and it is not the default, so it is not in the table above.
The barrier-frequency change is on by default with HK_LAZY_BARRIER=0 to
restore the old behaviour. The narrowed bit pattern is not on by default, and
that is deliberate: I can show it is correct for 16613 compute cases, and the
bits it drops were added for blits. Proving it safe needs a graphics
conformance run I have not done, so it stays behind a flag with the evidence
written down rather than shipping on a compute-only argument.
What is still open
Three things, stated as what they are rather than as a plan.
Nobody knows what bit 7 is. It is 14 of the 18 microseconds. Clearing it crashes the test suite but leaves llama.cpp decoding happily at 99 tok/s, which means whatever hazard it covers is one the decode path does not create. Naming it is the single highest-value piece of reverse engineering left on this driver, and it is worth roughly another 1.5x on this workload alone.
The marginal rate is 282 GB/s, 35% of the part's peak, and barriers are not why — with every barrier removed the effective rate does not improve, it is the fixed cost that disappears. The output head reaching 369 GB/s on the same hardware says the ceiling is higher than the layer matvecs are getting.
The gap to Apple's own driver is still about 2.5x. Metal on this same machine is reported around 160 tok/s for this model, which is ~380 GB/s, or 47% of peak. This work moved 52.4 to 64.8, which is 19.2%. Three compiler defects and one driver defect in, the part is still giving up most of its bandwidth, and the honest framing is that percentage rather than any ratio against another vendor's hardware.
One bit was the whole barrier
The sweep above walks cumulative prefixes — 0xffff, 0xff, 0x7f, 0x3f.
That is a clean single-bit delta only where two rows differ by one bit, which
holds for bit 7 and for nothing else. It cannot say which of bits 0–6 matter,
and the first version of this post read as though it could.
Clearing exactly one bit at a time from 0xff says it properly:
| clears | test-backend-ops | tok/s |
|---|---|---|
| nothing | 15937/16613 | 60.05 |
| bit 0 | 15937/16613 | 60.29 |
| bit 1 | 15937/16613 | 60.35 |
| bit 2 | 15937/16613 | 60.90 |
| bit 3 — usc_cache_inval | 15937/16613 | 60.09 |
| bit 4 | 13462/16613 | 60.34 |
| bit 5 | 15825/16613 | 60.04 |
| bit 6 | 12125/16613 | 60.46 |
| bit 7 | crash, no summary | 111.07 |
Bits 0 through 3 are each removable with no correctness change, and removable together. Bits 4, 5 and 6 are each load-bearing and each free — every arm sits at about 60 tok/s whether they are set or not — and they fail by very different amounts: bit 6 costs 3812 cases, bit 4 costs 2475, bit 5 only 112. Three independently required, independently free bits look like three distinct cache domains.
And bit 7 is the entire cost. Clearing it is +85%. Nineteen bits share 3.1% between them, and one bit has the rest.
That asymmetry is the interesting part. A bit that is required but costs nothing looks like an invalidate: drop some tags, no data moves. A bit that is required and costs 85% looks like a writeback: dirty data has to travel to the point of coherence and the barrier has to wait for it. Which is exactly the pair Mesa's own comment names — the PBE cache flushed and the texture cache invalidated. The blit fix set both classes; the ablation separates them.
Arithmetic that ties it off: 1/60.05 − 1/111.07 is 7.65 ms per token, spread over about 370 barriers, or 20.7 µs per bit-7 event against the 18.1 µs this post attributes to "a barrier". There was never a barrier cost. There was a bit-7 cost.
The ceiling was never there
Here is the part worth reading twice.
llama-bench reports tokens per second. It does not check them. Every rate in
this post measured on a crashing mask is therefore a measurement of how fast
the wrong answer arrives, and decode at bit-7-clear does not crash — it runs at
111 tok/s and looks perfectly healthy.
llama-perplexity does check. Same model, same corpus, same chunk count, only
the mask changing:
| mask | keeps | perplexity |
|---|---|---|
| 0x000fffff | stock, bits 0–19 | 2.4600 ± 0.12127 |
| 0x000000ff | bits 0–7 | 2.4600 ± 0.12127 |
| 0x000000f8 | bits 3–7 | 2.4600 ± 0.12127 |
| 0x000000f0 | bits 4–7 | 2.4600 ± 0.12127 |
| 0x0000007f | no bit 7 | inf ± inf |
| 0x00000080 | bit 7 only | inf ± inf |
So there is no 85% waiting to be collected. Bit 7 is required for the numbers to be numbers. The 2.08x further up stays true as an upper bound and was always labelled results-incorrect, but this is the measurement that proves it instead of assuming it.
There is also no cheaper primitive to reach for instead. The complete list of CDM packet types is Launch, Stream Link, Stream Terminate, Barrier and Stream Return; the machine description contains no wait, no fence, no semaphore. There is no scoped barrier on this hardware and no argument that narrows one. Which bits you set is the only gradation there is. The kernel-side barrier is a different layer entirely — a stamp wait between whole control streams, with no cache-domain fields at all, so it duplicates nothing and cannot substitute.
What that leaves is small and real: 0x000000f0, bits 4 through 7, is the
narrowest mask that holds. It passes the suite at 15937/16613 and returns
perplexity bit-identical to stock, and it is worth 58.51 to 60.30 tok/s,
+3.1%. It is not the default and should not be, because both of those gates
are compute-only, and the wide mask was written for blits. That needs a
graphics gate before it ships to anyone.
Credit
Honeykrisp is Alyssa Rosenzweig's and Asahi Lina's work, with Valve and Collabora behind much of it, and the AGX command stream in it was reverse engineered without documentation. The conservative barrier is not an oversight; it is a correct decision made with incomplete information, written down honestly with an invitation to revisit. All I did was accept the invitation and bring a measurement. The same applies to llama.cpp's Vulkan backend, whose alias tracking is what made it possible to ask how many barriers were actually needed.
