RDMA over a Thunderbolt cable: five driver bugs, a governor worth 1.87x, and the wall that is left
A verbs device built on USB4, between an M2 Ultra Mac Studio and an AMD Strix Halo box. Five layered driver bugs behind a permanent freeze, a CPU governor that silently reverted on reboot and was halving bandwidth by 1.87x, and a root-caused wall: every queue pair funnels through one transmit ring and one spinlock per rail. Then a size sweep that killed the reported cliff, a device advertising twice the message size it can register, and a 284B model at 90.18 GiB that one machine refuses in the allocator and both machines run over the cable. Plus eleven of the author's own numbers that later measurements demolished.
Two machines, one Thunderbolt 4 cable, and a kernel module that presents the cable to applications as an InfiniBand device. No switch and no NIC: the verbs device drives the USB4 host interface directly, so NCCL, MPI and perftest all work unmodified over a link that was built to carry a display and a disk.
The work here had three phases. Five driver bugs, layered so that each fix exposed the next, behind a run that froze permanently instead of failing. Then a CPU governor that had reverted to its default and was halving bandwidth. Then a scaling problem: adding queue pairs makes this driver slower, and the whole CPU bill stays flat while it happens.
Current state on this leg: 1.28 GiB/s on a single queue pair (11.0 Gbit/s), at 0.7 cores of CPU on the receiving machine. The link negotiates 20 Gb/s per direction. TCP over the same cable reaches 16.2 Gbit/s.
The two machines
Everything below was measured on this pair, on this cable. The values in this table are read from the machines, not from documentation.
The link runs at 20 Gb/s each way, which is the full Thunderbolt 4 rate. A “40 Gb/s” TB4 link is two lanes of 20 Gb/s, and dual-lane operation across domains is optional in the USB4 specification. AMD USB4 controllers do not negotiate it, so this leg is one lane per direction. Both ends report rx_speed=20.0 Gb/s tx_speed=20.0 Gb/s.
usb4_rdma0 and usb4_rdma5 over it, and every number below is bounded by that single lane.The freeze
The run that started this did not crash. It stopped. The client had completed send 13,242 and was waiting for a reply; the server had posted 13,241 replies. No RNR, no error, no duplicate, no timeout, and an empty dmesg on both ends.
The counters gave the answer. The server posted 13,241 replies, so it had received 13,241 requests. The client had sent 13,244 frames: 13,242 sends plus two retransmits. The duplicate-ACK counter read exactly 2, so both retransmits' originals had already been delivered and only their ACKs were lost. Request 13,242 was therefore acknowledged at the wire level and never delivered to the application. The interface had DMA'd the frame, told the sender it was fine, and never signalled the CPU.
A supplemental RX poll already existed for exactly this event. It was fully wired and hard-disabled by a one-line = false, because dual-sourcing the RX ring can reorder frames on the Apple verbs path; it is safe on the native path, where frames carry sequence numbers. Enabled behind a parameter, the first run rescued 1,288 stranded frames on one end and 9,383 on the other. One frame strands roughly every 280 under load, where the previous estimate had been one in 50,000.
Three more bugs followed. Every UC send leaked a send-queue slot, so the connection died at exactly max_send_wr = 1. A receiver-side watchdog errored the connection at 5.00 s while the sender's retransmit was due at 5.09 s, and its error-ACK then killed the sender's queue pair as well. And a tail fragment arriving before its missing head was treated as a protocol error and executed, rather than buffered until the retransmitted head arrived.
The clock
With the freeze gone, bandwidth came back noisy: the same configuration measured between 450 and 850 MiB/s across a day. The cause was not the fabric. Machine A's CPU governor had reverted to schedutil. A governor is a runtime setting, so a reboot restores the default and nothing in the health checks looked at it.
Same cell, same flags, one setting switched, run twice each way. The arms repeat to under one percent and do not overlap: 1.87x. The governor now has its own check in the leg-readiness gate, which refuses to measure while it is not set, because at schedutil every number is low by roughly that factor and nothing in the output says so.
With the clock fixed the same leg moved from about 0.85 GiB/s to 1.22–1.37 GiB/s on one queue pair.
The benchmark's own CPU
Two CPU measurements turned out to be about the measurement rather than about the transport.
ib_send_bw and ib_send_lat poll by default: the client spins on the completion queue. That spin competes with the same machine's frame delivery, which runs in a kernel workqueue, so the benchmark starves the work it is waiting on. One cell, both ends given identical flags:
ib_send_bw -c RC -s 262144 -q 1 -n 4000 -F # poll (default) ib_send_bw -c RC -s 262144 -q 1 -n 4000 -F -e # sleep on CQ events poll 611.43 MiB/s receiver 1.44 s CPU -e 827.20 MiB/s receiver 0.23 s CPU
The same trade runs the other way for latency, which is why events are not simply better:
ib_send_lat and ib_send_bw both default to poll mode, and the two goals want opposite modes. Spinning on completions wins latency by 2.2x; taking interrupts wins bandwidth. Neither arm is the honest one — the mode is part of the question being asked, and a number quoted without it is not a result.Polling gives 19.42 µs round trip typical against 43.14 µs event-driven, and it repeats to two decimal places. The event path costs about 24 µs per round trip. Neither mode is more honest than the other; they measure different operating points.
The second measurement problem was scope. The CPU accounting used systemd's figures for the unit running the benchmark, which is the benchmark process and nothing else. The work that moves frames is not in that process: the driver's pollers run in its own kernel workqueue and the receive path runs in softirq, which is charged to no process. Sampling the whole machine instead, over one 18-second window:
tbv_ibdev, accounts for 0.05 core-seconds — negligible — while the largest single term belongs to other tenants on the same host. A CPU claim here is a claim about somebody else's load, which is why none of the mode and governor comparisons above were taken from wall-clock CPU.The module's own workqueue accounted for 0.05 core-seconds in 18 seconds. The benchmark process was the largest attributable consumer, and the largest single term was other tenants on the same host.
Queue pairs
RDMA scales by adding queue pairs. On this driver it anti-scales: more queue pairs, less total throughput.
Independent processes, one queue pair each, separate ports and separate completion channels, sharing nothing in userspace — total throughput still falls as they are added. It reproduces at a small send window and at 64 KiB messages. Separating queue-pair count from send depth directly: one queue pair with 16 frames in flight reached 640 MiB/s, while two queue pairs with 8 frames each in flight — the same total depth through the same hardware — managed 274.
The source shows the mechanism. There is one transmit ring and one spinlock per rail, not per queue pair: path->tx_ring = tb_ring_alloc_tx(nhd, tx_hop, ...), with tx_lock taken at five or more sites in the transmit path. Every queue pair posts through that one point. Adding contenders adds latency and reordering without adding parallelism, and the cost appears as waiting rather than as CPU, which is what the flat CPU readings show.
Mellanox solved a version of this on the Ethernet side by moving from a single MMIO doorbell to multiple transmit and completion doorbells, and measured 9 to 56 million packets per second from that change. Here the hop index is a hardware resource bound to the cable and this leg has one cable, so the lever is making the single serialized post path cheaper per message rather than adding rings.
What the cable carries
iperf3, direct cable, 8-second runs A -> B, 1 stream 16.2 Gbit/s A -> B, 2 streams 16.0 Gbit/s A -> B, 4 streams 16.2 Gbit/s A -> B, 8 streams 16.4 Gbit/s B -> A, 1 stream 8.89 Gbit/s
Parallel streams add nothing, so one flow already saturates the fabric. 16.2 Gbit/s is 81% of the per-direction cap and is the practical bar for anything above the wire.
The gap that is available to close is the 1.4x to TCP, and it is software. The reverse direction runs at about half the forward one, 16.2 against 8.89 Gbit/s, and that asymmetry is not accounted for.
The cliff that was not there
A note carried forward from this work said the fabric had a hard ceiling above 256 KiB: RDMA WRITE peaked there, and a 1 MiB run produced no result row at all. That is the kind of finding that shapes a design, because it argues for chunking every transfer, so it was worth measuring rather than inheriting.
It is not a cliff. Sweeping message size with repeats at every point, sender on the Mac Studio, target on the Strix Halo:
The defect is an intermittent failure of roughly one run in five above about 1.5 MiB, not a boundary. That distinction is the whole point: a boundary is a limit to design around, and a one-in-five failure is a bug to fix. The original reading was a single sample of a probabilistic fault on a leg whose bandwidth noise floor is about 67%. 1 MiB now passes five times out of five, and 1.5 MiB failed twice and then passed twice inside the same ten minutes.
One hard limit did turn up, and it is worth reporting upstream on its own. ibv_devinfo advertises max_msg_sz of 0x1000000 — 16 MiB — while buffer registration fails with ENOMEM at 8. The advertised message size is twice what this stack can actually register, so a caller sizing its transfers from the device's own capability line gets a failure before a byte moves.
What the cable is for
A fast link is only interesting if something runs on it. The target is a model that does not fit on either machine alone, split across both by llama.cpp's RPC backend, with the split carried over verbs instead of TCP.
That needs a baseline first: the same model on each machine by itself. One file, 2.21 GiB, checksum identical on both ends, three backends. Every arm below ran through one harness that writes the model's checksum, the binary's build commit and the device line the run actually opened into the same log as the number.
Decode on the Strix Halo runs at twice the Mac's rate, and prefill at 3.45 times it. The middle bar is the one worth arguing with: on a single GPU under a single driver stack, Vulkan beats the vendor runtime by 14% on decode and 12% on a 2048-token prefill. “Use the vendor's runtime” is not a rule that survives being measured on this hardware.
The same server reports time to first token directly, which is not the same quantity as a prefill rate. A rate describes a pipeline that is already full; the first token is the one that waits for it to fill.
Strix Halo, Vulkan, llama-server timings.prompt_ms cold prompt, 1601 tokens 766.76 ms (2088 t/s prefill) prefix already resident 14.13 ms
Six identical requests look like a 54x spread until the response bodies are read properly: runs two through six each report prompt_n = 1, because the server caches the prompt prefix by default. One request processed 1601 tokens and the rest processed one. A spread that large between identical requests is almost never the machine, and this time it was the harness reading its own cache.
The RPC split engages. The Mac's server prints the line that proves it, and that line is the only admissible proof — llama.cpp's RDMA transport falls back to TCP silently whenever its probe fails, and because the QPN and PSN are exchanged over that TCP socket, the socket is present whether or not verbs ever carried anything. A working connection says nothing:
RDMA probed: dev=usb4_rdma0 gid=1 RoCEv2 qpn=6407 RDMA activated: qpn=6407->2700 mtu=4096 rx_depth=24
The first attempt never got past loading. At the transport's default queue depth of one, a 2.21 GiB model had not finished loading after eleven minutes. The device reports a maximum of four outstanding send work requests; at four, the same load completes and generates. That is a transport tuning fact rather than a property of the cable, and it changes the result from “does not work” to a number.
RPC over the cable, 37/37 layers offloaded prompt 208.7 t/s generate 47.1 t/s the same model on the Strix Halo alone prompt 2043.0 t/s generate 78.0 t/s
The layers really did cross the cable. What they crossed onto is the other machine's CPU. The Mac's RPC server is built without a GPU backend and announces 194,583 MiB of host memory as device memory, so the splitter — which places by free memory — hands it every layer. Offloading bought 44.4 to 47.1 tokens/s over running the whole model on the client's own CPU. Dragging a model across a cable so the far machine's CPU can run it is a loss against the GPU that was already sitting in the client.
So the transport question is answered and the placement question is not. A split across two GPUs needs a server built with a GPU backend on the far end. That is a build, not a limit of the fabric, and it is the next thing to measure.
A GPU on the far end
The build exists. The other machine in the fleet compiles llama.cpp's RPC server against Vulkan and libibverbs for x86_64, which is exactly the far end this needed, so the binary moved over the LAN rather than being rebuilt — 62 MB, SHA-256 identical at both ends, and it enumerates Radeon 8060S Graphics (RADV STRIX_HALO) on arrival. With that in place the split is GPU to GPU over verbs, and the placement question from the previous section has an answer.
Same model, same file, same harness as the CPU run — one arm on the Mac alone, one arm split across both machines over the cable:
Qwen3-4B-Q4_0, llama-bench, warm floor of repeated runs
TP=1 Mac Studio alone pp128 290.78 +/- 0.15 t/s
tg32 40.30 +/- 0.05 t/s
TP=2 both machines, RDMA pp128 464.46 +/- 85.33 t/s 1.60x
tg32 48.02 +/- 0.51 t/s 1.19xPrefill scales, decode barely does, and that is the expected shape: prefill is arithmetic that divides cleanly across two GPUs, while decode is one token at a time waiting on a round trip. The prefill error bar is large because it is — 85 t/s of spread on a leg whose measured bandwidth noise floor is 67%, reported rather than smoothed away.
But a 1.6x on a model that already fits is not why anyone builds a fabric. The real test is a model that fits on neither machine: a 284-billion-parameter checkpoint, Q2_K, 90.18 GiB resident. On the Mac alone it does not run at all — it fails before the first token, in the allocator:
Failed to allocate BO VMA BO creation failed llama_bench: error: failed to load model
Split across both machines over the cable, the same file loads and generates:
284B Q2_K, 90.18 GiB, Vulkan + RPC over usb4_rdma5 pp128 17.88 +/- 1.03 t/s tg32 3.51 +/- 0.12 t/s RDMA activated: qpn=6696->2600 mtu=4096 rx_depth=24
Those are not fast numbers and they are not supposed to be. The interesting quantity is not 3.51 tokens per second, it is that the alternative is an allocator error. This is the first result in the whole exercise where the cable is not an optimisation — the model has nowhere else to go. A fabric that makes a 90 GiB checkpoint runnable on hardware that individually refuses it is doing something a faster single machine cannot.
Two honest marks against it. The load transfer runs at about 146 MB/s while perftest reaches 900+ MiB/s on the same device — six times the available bandwidth left on the floor at model-load time, on a transport whose send ring is one deep.
The reason is not a missing knob, which is what I claimed here first and had wrong within the hour. GGML_RDMA_SEND_DEPTH exists and accepts 1 to 512 at runtime. What limits it is pinned memory: the default receive ring is 24 chunks of 256 KiB, the chunk size is chosen in the source with the comment “fits default 8 MiB memlock”, and this host's RLIMIT_MEMLOCK is 8 MiB with the hard limit equal to the soft one, so an unprivileged process cannot raise it. Six MiB of receive ring leaves about two for transmit, which is a send depth of eight before registration fails. The lever is real, it is reachable without a rebuild, and it is boxed in by a limit one line away from the knob.
The second mark is worse than it first looked. The process aborted with double free or corruption (out) during teardown, after printing its results — and then the Mac hard-reset. Its journal stops mid-sentence at 22:39:00 on an ordinary disk-telemetry line, with no shutdown sequence, no panic, no OOM kill and nothing in pstore; the machine was back up at 22:40:58. That is the second unexplained reset on that host the same day. Through the run the link had been retransmitting steadily — native ACK matched after retry … rnr_retries=1, once every few seconds — which is the same large-message retransmit signature the size sweep found and did not explain.
I cannot show that the benchmark caused the reset, and two minutes separate them. But the numbers above were produced by a run that ended with a corrupted heap and a machine that went down without saying why, and reporting the tokens per second without that sentence would be the twelfth entry in the corrections list rather than an absence of one.
What the file actually contained
Every number above was taken after checking that both machines held the same model file. That check found something else on the Mac.
Three reads of one inode, with nothing writing to it and the modification and change timestamps unchanged since the file was created, returned three different checksums. Reading the same file with O_DIRECT — which bypasses the page cache and goes to the device — returned the same correct checksum every time, and it matched the independent copy on the other machine.cmp against a direct-read copy put the difference in a single contiguous run of 397 bytes at offset 1,939,111,937.
buffered (page cache) e13a4d3b0b1d5ff00858de695be36d90 device (O_DIRECT) ff009625971c56242d3f046ab7f53fc7 <- correct after evicting that one file's pages with posix_fadvise(DONTNEED) buffered (page cache) ff009625971c56242d3f046ab7f53fc7 <- agreed, and stayed agreed
The disk is fine and so is the read path. The copy resident in memory had been altered. Eight rounds of evict-and-reread on an idle machine were clean every time; three short Vulkan runs produced one corrupted result with a fourth distinct checksum. A different wrong value each time is what separates an active writer from a broken read path — a filesystem mis-decompressing an extent would produce the same wrong bytes on every attempt, and this never did. The mechanism is not identified here and no driver is accused.
llama.cpp memory-maps the model, so a load reads exactly those pages. A model can run on weights that are silently not the weights on disk, with no error, no counter, and no line anywhere — and non-deterministically, because it depends on what is in cache. Throughput is unaffected: the same arm measured 591.30 and 591.62 prefill while the cache was in flux and after it was healed, inside the noise. Any claim about output quality from this machine would not be safe. The harness now digests the model both ways on every arm, heals once by evicting, and refuses if the two paths still disagree — it caught the corruption again on the very next run.
Corrections
Eleven numbers from this work were published and later withdrawn. They are listed here because the pattern in them is more useful than any of the individual results.
Every one of those was a measurement whose parts were not measured over the same thing. A ratio with numerator and denominator from different windows. A process's CPU read as the transport's. A benchmark's spin read as a wire cost. A specification read as a measurement.
Where this stands
The freeze is fixed and the run completes. Throughput is 1.22–1.37 GiB/s on one queue pair, 60–70% of what TCP achieves on the same cable. Latency is 18.29 µs round trip typical and 11.74 µs at its floor, with a 24 µs event wakeup that polling hides rather than removes.
The open problem is queue-pair serialization. It is why a multi-GiB/s target is not reachable on this cable yet, and it is not the wire, the CPU, or a setting. It is a change to the transmit path.
The question this post ended on last time was whether a GPU-backed RPC server on the far end makes this a fabric for compute or only a very fast way to move bytes. It is answered, and the answer is neither of the two things the question offered. On a model that fits, the split buys 1.60x on prefill and 1.19x on decode — real, modest, and not a reason to own a second machine. On a model that does not fit, the split is the difference between 3.51 tokens per second and an allocator error. The value of this cable is not throughput. It is capacity, and capacity is the one thing a faster single machine cannot be bought into.
Community measurements on the same hardware class: Level1Techs USB4 clustering notes. The module: thunderbolt-ibverbs, including its tuning sweep. Doorbell contention: the mlx5 multiple-doorbell series.
