Thunderbolt RDMA: 8.4 microseconds, of which about one is the cable
A direct USB4 cable between a Mac Studio and a MacBook carries RDMA at an 8.4 us floor that does not move with message size, and a receive ring that admits exactly floor(1024/frames_per_msg) - 1 messages. The cable trained at 40 Gb/s. Almost none of the cost is the cable.
There is a cable running between a Mac Studio and a MacBook Pro on my desk. The Mac Studio runs Linux, which is only true because last week I spent a weekend hauling Apple's USB4 host controller out of kernel 7.2 and into 7.1.13, where the Asahi GPU driver still works. Asahi's 7.1 branch builds the GPU and has no USB4 at all; 7.2 has USB4 and marks the GPU driver BROKEN. You get one. I wanted both, so about 1,100 lines moved backwards.
Apple's position is that this machine cannot do any of what follows. RDMA over Thunderbolt, on macOS, requires Thunderbolt 5 and an M3 Ultra, M4 or newer. A 2023 M2 Ultra has Thunderbolt 4 and is not on the list. It is doing RDMA anyway, because it is not running macOS.
This post is what the cable does once RDMA is running over it. One number is worth the whole exercise: the one-way latency floor is 8.4 microseconds, and it does not move when the payload goes from 2 bytes to 64. Roughly one of those microseconds is the wire. The rest is software I did not write.
The kernel module is hellas-ai's thunderbolt-ibverbs and their writeup is the reference work. They did Mac RDMA over Thunderbolt before I did. What I have that they do not is Apple Silicon on the Linux side of the cable, which is the entire reason the backport was necessary. If you want the verbs vocabulary — queue pairs, work requests, credits — NVIDIA's programming guide is better than anything I would write here.
Test bench
Three machines, two cables, no switch. There is no PCIe link table because there is no PCIe in the path.
Trust the module's own bind line for what the link negotiated, not ethtool:
Two lanes at 20 Gb/s is 40 Gb/s per direction. ethtool prints a flat
40000Mb/s on the tbnet interface whether or not that is what trained. It is
a constant in the driver, and quoting it as evidence would be a mistake.
The floor
Before RDMA, plain IP over the same cable, 200 packets, warm:
| path | min (ms) | mean (ms) | max (ms) | mdev (ms) |
|---|---|---|---|---|
| A to C over the USB4 cable | 0.055 | 0.314 | 0.775 | 0.179 |
| A to C over the LAN | 2.088 | 2.771 | 10.968 | 0.819 |
A 55 us floor with a 314 us mean, on a link with no switch and no contention, already says the transport is faster than whatever sits above it.
RDMA puts a number on that. The one-way floor on the Apple-to-Apple link is 8.4 to 9.9 us, flat from a 2-byte payload to a 64-byte one. Real RDMA hardware does small messages in 1 to 2 us. The gap is not mysterious, and you can read it off the device before timing anything:
One queue pair. No inline data. No atomics. This is a software provider wearing the verbs API, and every send walks the kernel and Apple's ACIO coprocessor firmware instead of being handed to a DMA engine that owns the doorbell. A cost that does not change with message size is fixed overhead by definition; serialising 64 bytes onto a 40 Gb/s wire is around a microsecond of the 8.4.
Scroll horizontally to see the full diagram.
Approximate cost estimates, not a traced timing breakdown. Sweep endpoints are schematic; raw per-size samples are not published in this post.
The receive ring, which is the actual find
Latency is a cost per message. Throughput exposed something better. Running
payload-checked UC SEND from host A to host B (uc_oneway --check, every run
data_ok=yes, zero mismatches), the receiver asks for a 512-deep receive
window every time and never gets it:
| message size | rate (Mbit/s) | recv WRs accepted | frames/msg | frames in window | result |
|---|---|---|---|---|---|
| 16 KiB | 2505.58 | 255 | 4 | 1020 | ok |
| 64 KiB | 3129 - 4402 | 63 | 16 | 1008 | ok |
| 128 KiB | 3790 - 3813 | 31 | 32 | 992 | ok |
| 256 KiB | 3869 - 4639 | 15 | 64 | 960 | ok |
| 1 MiB | — | 3 | 256 | 768 | FAILS |
What it gets instead is exact:
Five sizes across six doublings, no residual. Not a budget hovering near a thousand — a 1024-frame ring with one slot held back, which is the ordinary way to tell a full ring from an empty one. The window counts frames, not work requests, so a bigger message buys you fewer messages in flight and nothing else.
Scroll horizontally to see the full diagram.
Measured window totals. accepted = floor(1024/frames_per_msg) - 1 at every size.
That is also the shape of the throughput curve. It climbs from 16 KiB to 64 KiB as per-message overhead amortises, then flattens near 3.9 Gbit/s because the ring will not let the pipeline get deeper. Peak is about 10% of what the link trained at, and the ceiling is the credit window rather than the cable.
I got this table wrong the first time I published it. The 256 KiB row said
1008 frames. Fifteen 64-frame messages is 960. I had not multiplied anything —
two recv window: lines had not printed, and I filled the gap with a number
that continued the visible pattern (1020, 1008, 992). It looked right, which
is the problem with numbers that continue a pattern. Re-reading the runs gave
960, and the exact formula only appeared once the wrong number was gone.
Four bugs, two of which are worth your time
Host A had a working link to host B. Adding host C on a second cable failed four times before the two ends would even shake hands. Two of those failures generalise well past this driver; two were local mess. I am telling you the two and listing the others.
A module-wide parameter for a per-device property. The second cable's peer sent HELLO and it matched nothing, because the new device was advertising the first cable's address. The module takes its RoCE GID from a named netdev:
One charp, module-wide. Two cables means two tbnet interfaces and two IB
devices, both reading their address out of that single string. The fix was
already in the source one function down — the Apple backend has its own
override, tbnet_identity_gid, defaulting to auto, which falls back to
roce_netdev. The fallback is what hid the collision: with one cable it is
always correct, so the bug cannot exist until your second link. Setting both
explicitly separates them:
I would rather this parameter had been per-device from the start, but I also understand why it was not: nobody writes the two-cable case first.
A device that is ACTIVE and invisible. On host C, rdma link show listed a
healthy ACTIVE device, /dev/infiniband/uverbs0 existed, the provider was
installed and registered — and ibv_devices printed an empty table. No error.
The provider's own source comment predicts exactly this:
Fedora's persistent-naming rule had renamed the device to rocep199s0f6. With
no upstream driver enum to match on, the provider falls back to matching the
name prefix, and the prefix was gone. Renaming it back brought the device
straight into ibv_devices. That is a runtime fix, so it needs a udev rule
keyed on the kernel name to survive a reboot — keying it on the node GUID, as I
did at first, only excludes the one device you already know about.
The other two, briefly. NetworkManager kept handing the interface a
169.254.x address alongside the static one, and the module takes the first
IPv4 address it finds; ipv4.method manual ends it. And ib_send_lat failed
on both ends at once with a socket error that was really a build mismatch —
both binaries reported Version: 6.29, one was the distro package and one was
a source build, and they disagreed on the negotiation struct. Same version
string, different wire format. Build both from source.
One more practical note. The module's parameters have ordering constraints and
it reports them one at a time, so you find a working configuration by walking a
chain of refusals: negotiate_native=1 requires start_rings=1 requires
allocate_rings=1; tbnet_identity=minimal_packet wants XDomain
unregister-drain support that the backported 7.1.13 does not have, so
stock_proxy is the identity that works there. Budget several load attempts.
The configuration that brings both links up:
One direction still does not work
Handshake completes, transfer stalls. The debugfs counters say where, without any guessing:
Host C posted 512 sends and completed all 512; host A received all 512. That direction is fine, which clears the cable, the tunnel and the negotiation. The other way, host A posted 529 and completed 15, while host C completed zero receives and returned zero RX credits. Host C never finishes a receive descriptor, so it never returns credits, so host A spends its opening allowance and stops. The 15 are that opening grant, not a partly working path.
Two lines from host C's log are the leading suspects and neither is confirmed:
-12 is ENOMEM on the second rail's probe. Rail 0 registered fine and is the
rail carrying this traffic, so the link between a failed rail-1 probe and a
dead rail-0 receive path is precisely what has not been established. Plain IP
over that same cable runs both directions at 14 Gbit/s, which puts the fault in
the ibverbs data ring rather than the link or tbnet. I have a suspect and no
evidence, so it stays a suspect.
Numbers next to other people's numbers
| measurement | hellas-ai (published) | this bench |
|---|---|---|
| one-way latency, small message | ~7 us (ib_write_lat, 64 B, 1 QP) | 8.4 - 9.9 us (ib_send_lat, 2-64 B) |
| bandwidth, per direction | ~48 Gb/s (1 MiB, 8 QPs, 4-HCA aggregate) | ~3.9 Gb/s (1 QP, 1 cable) |
| bandwidth, bidirectional | ~95 Gb/s | not measured |
| Mac endpoint | 6.5 Gb/s Mac->Linux, 1.8 Gb/s Linux->Mac, ~10 us RTT | 4.4 Gb/s Linux->Mac, 0.226 ms RTT (IP) |
Their ~7 us against my 8.4 to 9.9 is the useful agreement here. Two setups
that share no hardware landing in the same range is far better evidence that
the floor is structural to the software path than either measurement alone. The
bandwidth gap needs no explanation: they aggregate 8 queue pairs across 4 HCAs
and two cables, and my device reports max_qp 1.
Apple already ships this, and it is called JACCL
The first version of this post implied nobody had published Mac-to-Mac RDMA
numbers. That was wrong, and the thing I missed is not obscure: Apple shipped
it at WWDC 2026. JACCL — Jack and Angelos' Collective Communication Library
— is an MLX distributed backend that runs collectives over RDMA on Thunderbolt 5,
and it is the reason rdma_ctl enable exists on the macOS side of my own cable.
What Apple's own MLX documentation commits to is one qualitative sentence: JACCL achieves "communication latency an order of magnitude lower than the ring backend". The figures everyone repeats — single-digit microseconds against roughly 300 for the TCP ring, 50 to 60 Gb/s, about 3x on four M3 Ultras fine-tuning — come from the WWDC session and get recycled through every write-up since.
The people actually running it have published better material than the write-ups have, and it is all in the MLX discussions rather than on anybody's blog.
#3481 builds a file-transfer
tool on mx.distributed.send/recv and sustains 3.5 to 3.8 GB/s over TB5 —
16 files, 49.63 GB in 13.0s. The comparison inside that post is the part I care
about: rsync over the same Thunderbolt cable, running IP instead of RDMA,
manages 300 to 500 MB/s. Same wire, roughly eight times the throughput, and the
only thing that changed is how much software sits in the path. That is this
post's whole thesis, measured by somebody else on hardware I do not have.
It also carries the sort of detail that only comes from doing it. send and
recv default to the GPU stream; Metal has an unconfigurable command-buffer
timeout of about five seconds, and an RDMA receive that outlives it takes the
process down — so every call needs stream=mx.cpu. JACCL needs an all_sum
barrier immediately after init(), and it has to be mx.ones(10), because the
single-element version does not reliably trigger the full synchronisation path.
Protection domains leak on teardown, so roughly sixty init/teardown cycles
exhaust them and the only recovery is rebooting the node.
What still does not exist anywhere I can find is a per-operation latency
number for JACCL. No ib_send_lat, no invocation, no config printed beside a
microsecond figure. Bandwidth, yes. Tokens per second, yes. The one number this
post is about, no. If you have one, send it.
So the comparison is worth setting out honestly, including which column is a measurement and which is a vendor claim:
| JACCL (Apple) | hellas-ai | this bench | |
|---|---|---|---|
| small-message latency | "single-digit us" (claim) | ~7 us (measured) | 8.4 - 9.9 us (measured) |
| published invocation | none found | yes | yes |
| link | Thunderbolt 5 only | USB4 40 Gb/s | USB4 40 Gb/s |
| both ends | macOS, M3 Ultra / M4+ | Linux x86 + Mac | Linux on M2 Ultra + macOS |
| topology | fully connected mesh or ring | 2 cables, 4 HCAs | 1 cable, 1 QP |
Three numbers in the same band is the interesting part. Apple's native stack, hellas-ai's out-of-tree module on Strix Halo, and a software verbs provider on a Linux Mac all land within single-digit microseconds of each other. If a vendor's own RDMA path and a third-party shim are that close, the floor is not being set by anybody's implementation.
So what is actually mine? Narrower than I first wrote, and JACCL is why — but also stranger than I first wrote, in a way worth being precise about.
JACCL needs Thunderbolt 5 and an M3 Ultra or M4-class chip. This Mac Studio is a
2023 M2 Ultra with Thunderbolt 4. By Apple's own requirements it is not a machine
that does RDMA over Thunderbolt — not slowly, not in a degraded mode, at all. The
rdma_ctl enable path does not apply to it.
It is doing 8.4 microseconds anyway, one cable, no switch, because the operating system on it is not the one carrying the requirement. The silicon was evidently always capable; a support matrix is a support matrix. So the defensible claim is: Apple Silicon as the Linux endpoint, over Thunderbolt 4, on a machine excluded from the vendor's own feature, with every command printed.
Everybody publishes tokens per second. Nobody publishes the cable
This is the actual gap, and it took being wrong about it twice to see it.
exo have done the most work here by a distance, and say themselves they spent the past year working with Apple on RDMA networking over Thunderbolt 5. Their four-Mac-Studio cluster — 1.5 TB of unified memory, about forty thousand dollars — runs Kimi K2 at roughly 25 tokens per second with around 2 seconds to first token under 450 watts, Qwen3 235B at 32, DeepSeek V3.1 671B at 32.5. Those are real, useful, reproducible numbers, and they are all end to end. None of them tells you what the interconnect contributed.
Their DGX Spark plus Mac Studio piece is the sharpest example, and it is good engineering: prefill on the Spark (around 100 TFLOPs FP16), decode on the M3 Ultra (819 GB/s of memory bandwidth), KV cache streamed between them layer by layer. Llama-3.1 8B goes from 6.42 s on the Mac alone to 2.32 s combined. The link carrying that KV cache is 10 GbE, quoted at its nominal speed and never measured. A Thunderbolt cable between those same two machines trains at four times that, and nobody has published what it does.
The state of the public record, as far as I can find it:
| source | what was measured | the link | link measured? |
|---|---|---|---|
| exo, 4x Mac Studio | tok/s end to end, 3 models | Thunderbolt 5 RDMA | no |
| exo, Spark + Mac Studio | prefill/decode split, 8B | 10 GbE (nominal) | no |
| Apple, JACCL | an order-of-magnitude claim | Thunderbolt 5 RDMA | no invocation published |
| MLX #2990, 5x M3 Ultra | PP vs TP tok/s, Kimi-K2 1T | TB5 full mesh, RDMA | no |
| MLX #3481 | 3.5-3.8 GB/s file transfer | TB5, RDMA vs IP on one cable | yes, throughput |
| corti, Mac + Spark via exo | tok/s end to end | 1 GbE, called the bottleneck | no |
| hellas-ai | ib_write_lat, ib_write_bw | USB4 40 Gb/s | yes |
| this bench | ib_send_lat, uc_oneway --check | USB4 40 Gb/s | yes |
The 1 GbE row is corti's Mac Studio plus DGX Spark cluster, which is honest about it — the post says outright that the network is the bottleneck. It is still the most detailed public writeup of that pairing, and it is running over one gigabit while a Thunderbolt cable sits on the same desk.
There is nothing wrong with an end-to-end number; it is what you actually get. The problem is that when the tokens per second disappoint, an unmeasured link is the first thing blamed and the last thing checked. That is the hole this post fills, and it is a small hole beside what exo have built.
What I would measure next
Three limits, three different experiments, none of them done:
The 8.4 us floor should not be quoted as a hardware limit. It is what a software verbs provider costs on this path today, and a kernel trace has to split the driver path from the ACIO firmware round trip before either gets a number.
The 3.9 Gb/s ceiling belongs to one queue pair against a fixed ring. More queue pairs is the obvious next throughput experiment. Payload-checked transfers up to 256 KiB and reproducible failure at 1 MiB give it a concrete boundary to push at.
The host A to host C failure needs receive-ring evidence on host C during the
stall, and an answer to whether the failed rail-1 probe touches rail 0 at all.
Until then the ENOMEM is a lead.
And then the thing this was all for: put a model across the cable. This one has a published prior, which makes it a much better experiment than it was an hour ago.
MLX discussion #2990 runs exactly this comparison on five M3 Ultras in a full TB5 mesh, on Kimi-K2-Thinking at 1T parameters, and finds that pipeline parallel and tensor parallel land on top of each other:
| config | nodes | tok/s | load time | peak memory |
|---|---|---|---|---|
| PP5 | 5 | 14.45 | 5.2 s | 102.5 GB |
| PP4 | 4 | 14.49 | 22.3 s | 145.9 GB |
| TP4 | 4 | 14.82 | 99.9 s | 185.7 GB |
| TP2 | 2 | 13.15 | 92.6 s | 345.0 GB |
| TP5 | 5 | failed | — | 12288 does not divide by 5 |
Their conclusion is that a TB5 ring will likely beat a fully meshed TP cluster without the cable spaghetti. And note what makes that true: the collective is nearly free. On JACCL it is.
It is not free here. This fabric charges 8.4 microseconds per message no matter
how small, and admits floor(1024 / frames_per_msg) - 1 messages at a time.
Tensor parallel is many small synchronous all-reduces, which is precisely the
traffic shape that fixed per-message cost punishes hardest. Pipeline parallel is
fewer, larger, asynchronous exchanges — and larger messages here buy fewer slots,
which cuts the other way.
So the prediction, written down before the run so it can be wrong: the 2.3% gap should open up substantially on this link, in pipeline parallel's favour. If it does not, my model of the ring is incomplete and that is worth more than being right. llama.cpp first because it is already here — tensor parallel split by row and by column, then layer split, then pipeline. Same model, same prompt, same harness, warm floors.
Further out, and only if those numbers earn it: MLX on Linux over Vulkan, and possibly dusting off my own DSL-style LLM runtime, which would let me put each collective exactly where the ring law says it should go rather than where a framework happens to put it. That is a lot of work to justify with a benchmark that does not exist yet.
Further out, and only if the llama.cpp numbers earn it: MLX on Linux over Vulkan, and possibly dusting off my own DSL-style LLM runtime — which would let me put each collective exactly where the ring law says it should go, rather than where a framework happens to put it. That is a lot of work to justify with a benchmark that does not exist yet.
Two measurements that do not exist publicly, if anyone wants them: an independent per-operation benchmark of JACCL with its invocation printed, and a Mac-to-DGX-Spark link measured with perftest. Both are worth more than anything in this post.
