Grouped hnorm on mlx-serve: +41% accepted tokens per round, and a second tenant on fedora
A one-line change to how mlx-serve groups RMSNorm scales raised speculative-decoding acceptance from 55.6% to 64.7% (+41% accepted tokens per round) on an M2 Ultra, with quality unchanged. Plus: an on-demand serve measured at 31.6 tok/s code / 247 prefill@16k running beside the live serve on one Strix Halo.
We changed one thing in how mlx-serve normalizes activations, and speculative decoding accepted 41% more draft tokens per round: acceptance went from 55.6% to 64.7% on an M2 Ultra. Decode throughput rose 4.7–19.3% depending on model and state. Quality was unchanged — same pass rate on the code suite with the change on and off. It is now the default in the production serve on that machine.
Background, in three sentences
Speculative decoding generates text with two models: a small fast draft model proposes several tokens, and the big model verifies them in one pass. Acceptance is the fraction of proposed tokens the big model keeps — higher acceptance means fewer wasted verification passes and faster decoding. RMSNorm is the layer that rescales activations; the patch changes only how its scales are grouped.
The numbers
M2 Ultra, A/B, grouped hnorm off vs on acceptance 55.6% -> 64.7% accepted tokens/round +41% decode throughput +4.7% .. +19.3% (depends on model + state) code suite (HumanEval N=40) green on both arms
How the comparison was run
The patch was staged env-gated with the default off (1878d35), so the “off” arm runs the exact upstream code. A script owns the whole decision: it locks the GPU so nothing else runs, does 3 fresh process starts per arm (throughput is noisy per start), runs the code suite on both arms, and promotes the patch only if it wins on speed with no quality loss. It won; the promotion to the production pack (ee91bed) was that script's decision, not a manual flag.
One detail worth noting: acceptance is usually treated as a property of the model checkpoint. Here it moved under a runtime change with the checkpoint untouched — so it is partly a property of the serve. The quality suite is what tells “better drafting” from “easier-to-guess target”: a damaged target would accept more drafts and fail the suite. It passed.
The second result: two servers, one chip
The same week, our Strix Halo box (fedora) — which already runs the live fast-brain serve — ran a second, on-demand model server beside it: 31.6 tok/s decoding code and 247 tok/s prefill at 16k context, measured while the live serve kept running. Both servers share one memory pool, so these are co-tenant numbers — the condition a small fleet actually runs in, not a quiet-box best case.
Together the two results say something practical for small fleets: you can A/B a serving change on the box that runs production, and you can run a second model beside a live one — if you report the numbers as measured, together rather than alone.
Reproduction
Full logs and the raw ladder artifacts live in the horde repo (docs/horde-knowledge and the go-live handoff). This post is the summary; the artifacts are the evidence.
