A Thunderbolt packet switch with no incarnation counter, and the five bugs it hid
A UC verbs pingpong across a Thunderbolt 4 cable froze permanently on every run, with clean counters on both ends and empty dmesg. The frame had been DMA'd and acknowledged on the wire; the CPU just never heard about it. Five layered driver bugs sat behind that one symptom, and each was named by a counter before it was fixed.
The previous post on this bench ended on a stall it could not explain: one direction worked, the other completed zero receives, and the leading suspect was a failed rail-1 probe. That suspect was wrong. The one-way stall came from two module configurations resolving the same default differently — one end turned end-to-end flow control on, the other turned it off, and the end that turned it on sat waiting for credits the other had no idea it owed. One explicit parameter fixed it.
What follows is the longer one, because fixing it removed the only thing that had been hiding a worse failure. The disabled reaper came first; four more bugs sat behind it, each one visible only once the previous fix let the run get further.
Test bench
Both hosts are soldered-SoC machines: there is no PCIe in the path and no BIOS. The Mac's TB4 ports are on the Apple NHI (0xd004c throttle register, programmed by the kernel at ring activation); the Strix Halo's are on the stock Intel-style NHI (0x38c00, written directly by the module). That difference is load-bearing for two of the bugs below.
The freeze
The symptom, run after run:
No RNR, no duplicate, no error, no timeout. dmesg was empty on both ends, and
both processes were alive and polling. A UC pingpong at 4 KB froze permanently,
most runs, at points between a few thousand and a few tens of thousands of
iterations.
The first suspects were the cable and the switch. A Thunderbolt link is a
packet switch, and the spec
gives it no message-level sequence number to work with. Upstream
PR #44 had already
established that the Apple TX engine cannot distinguish overlapping sends on
one queue pair, and serializes UC sends for that reason. I had that patch's
workaround in place (native_tx_max_inflight=1), and it moved the freeze
point roughly 200x — from a few hundred iterations to tens of thousands. It
did not remove the freeze.
One number broke the hardware theory open. refuted The client had posted 13,242 sends
and the server had posted 13,241 replies. The client's own counters said
data_wr_retransmit: 2. Those two retransmits were accounted for: two earlier
epochs, each recovered. So send number 13,242 was never retransmitted, which
under the reliability design meant the client had proof of delivery for it —
an ACK.
The receiver acknowledges a UC send only when the whole message is assembled and delivered into a receive buffer. So an ACK that reached host A means host C's driver did complete the frame at the ring level. But the server application had one fewer request than the client had sends. Both facts are true simultaneously only if the frame's completion was consumed at the ring and lost between the ring and the verbs receive queue.
That is a stranded completion: mailbox DMA'd, wire ACK sent, CPU interrupt never delivered (or delivered and dropped), frame sitting in a completed descriptor that nobody reaped.
The disabled RX reaper
The module already contains a supplemental RX poll: a delayed work item that reaps completed descriptors off the RX ring in case the interrupt path missed them. It is armed after every TX post, with a 1 ms delay and a 16 ms window. It was correct, and it was off — a single line in path initialization, with a comment explaining that dual-sourcing RX could reorder frames on the Apple-compatible verbs path, which carries no per-message sequence number.
That comment is right about the Apple path. It is wrong about the native one:
native frames carry PSNs and the native receive path serializes on a per-QP
lock, so a second reaper cannot expose frame N+1 before frame N. The poll now
enables itself on native paths only, behind a module parameter
(rx_supp_poll=-1, auto; 0 off; 1 on everywhere).
The first run after enabling it counted 18,334 rescued frames on host C and 9,383 on host A, in a single million-iteration run. measured Roughly one frame in 280. The previous estimate in my notes was one in 50,000, off by two orders of magnitude. It had been measuring the true wire drops instead, and attributing every loss to them.
The rescues are visible in the counters, and none of them cost a retransmit:
Four more bugs
With the freeze gone, the run got further and died differently.
The send queue slot leak. Run two died at 179,490 sends with the app
printing "Couldn't post send" — and zero counters anywhere. Every
unsignalled UC send was leaking one send-queue slot: the completion path
released slots only on teardown and on error flushes, never on the success
path that a UC queue pair actually takes. ibv_uc_pingpong creates its QP
with cap.max_send_wr = 1, so a leak of one slot is fatal immediately; the
question was only when. The slot now releases at the local drain — the point
the send queue is actually empty — not at ACK time, which is ~70 µs too late
for a depth-1 queue.
The watchdog racing its own retransmit. Run three died at 40,484: a 4 KB send is two native frames (4048 + 48 bytes) and the tail vanished. The receiver's active-message watchdog errored the half-assembled message 90 ms before the sender's retransmit was due (5.00 s vs 5.09 s), and its error ACK then killed the sender's queue pair too. The receiver now out-waits the sender's entire retransmit budget instead of racing it.
The orphan fragment. Same run, different case: when the head fragment vanishes and the tail arrives first, the frame is perfectly recoverable — the sender will retransmit the whole message at the 5 s mark. The old code treated a tail at offset 0 as a protocol error and killed the connection. It now buffers the orphan through the reorder machinery and waits.
The RNR dead-end. An RNR (receiver-not-ready) ACK on a UC queue pair froze
the send forever: RNR retries were gated on the send being "retryable", which
was only ever true for RC. UC queue pairs carry no rnr_retry value, and zero
was being read as "zero retries" rather than "use the module's budget". Same
policy pattern as the wire-retransmit budget: a UC send now gets a module-level
RNR budget.
Every one of these was diagnosed from counters rather than from reading code.
When a run freezes, read /sys/kernel/debug/thunderbolt_ibverbs/peers on both
ends and do the arithmetic. The counters are per-end and per-ring, and they
decompose a run exactly.
The acceptance gate
| measurement | result | method |
|---|---|---|
| 1M UC pingpong, 4 KB payloads | 96.19 s, 681.33 Mbit/s | ibv_uc_pingpong -c (checksummed), rc=0 on both ends |
| 64 B latency floor | 57.64 us | 10 x 1M-iteration runs, converged, warm floor |
| 16 MiB byte-verified echo | 1.049 Gbit/s | full_echo_equal=true, my own harness |
| MLX ring all_sum, 64 MiB | 0.33 Gbit/s, CORRECT | MLX ring over the same link |
| recovery accounting, the 1M run | 18334 rescues, 0 retransmits, 0 errors | debugfs counters |
What is still wrong
Three things, stated because a post that only lists wins is a brochure. open
Bandwidth is poor. 1.049 Gbit/s on a 20 Gb/s link is not RDMA-grade, and 0.33 Gbit/s on the collective is worse. The cost is per-frame completion: a 16 MiB transfer is 4,096 frames of 4 KB, each paying a completion and a copy. Bigger native frames (or a striped multi-frame path) are the obvious fix, and it is not built yet.
A truly lost frame costs ~5 seconds. Recovery works, but it costs a retransmit timer. A selective fragment NAK — asking for just the missing piece — would cut that to a round trip. Not built.
The loss itself is not gone. The switch still drops roughly one frame in a few hundred under load; what changed is that the driver now notices and recovers instead of hanging. That rate looks like a property of this particular Apple NHI + cable combination at 20 Gb/s, and I would want a second cable and a second machine pair before calling it a property of the design.
Receipts
All patches, both modified trees, the counter-forensics log, and the harness
are in the horde repo (lane/tb4-fabric-benchmark): the two kernel patches are
patches/tbv-uc-local-completion.patch and patches/tbv-rx-supp-poll.patch,
and build-lanes/lanes/tbv-uc-scale-stall.md carries every counter quoted
above. The module tree with the patch history is mirrored at
afrog33k/thunderbolt-ibverbs
(and on the fleet's own gitea as ron/thunderbolt-ibverbs, branch
lane/tb4-fabric-benchmark). The fixes are param-gated and native-only, so a
PR series upstream is the next step, not a rewrite.
