Four GPUs with no peer-to-peer: a kernel allow-list and an IOMMU domain

Four AMD Radeon AI PRO R9700s moved from an EPYC board to an interim dual-socket X99 board while the EPYC was replaced. Same cards, same ROCm, same checkpoints. On the EPYC they had all-to-all peer-to-peer; on the X99 they had none at all, and a custom two-rank all-reduce died with hip: invalid device pointer. The cause was not the cards, the firmware, or the cabling. It was one branch in one Linux kernel function, plus a second gate nobody hits until the first one passes. Fixing both took two lines of kernel source and one boot flag, and prefill throughput went up 85–91%.

Test bench

This is an interim board — the production EPYC was out for replacement — and that context explains most of what follows. An entry-level CPU in a four-GPU chassis is a real confound, and so is a riser.

motherboard   MACHINIST dual-socket X99 (Intel C612), AMI Aptio V
CPU           2x Intel Xeon E5-2620 v4 (Broadwell-EP), 8C/16T each
              16C/32T total, 2.10 GHz base / 3.00 GHz max
memory        377 GB DDR4. DIMMs rated 2400 MT/s, CONFIGURED 2133 MT/s
              (the E5-2620 v4 caps there -- configured is the number that counts)
GPU           4x AMD Radeon AI PRO R9700, 32 GB each, gfx1201 / RDNA4
              05:00.0  09:00.0   (socket 0 / NUMA node 0)
              86:00.0  89:00.0   (socket 1 / NUMA node 1)
NIC           Mellanox ConnectX-4 Lx, 25 GbE, RoCE to a second host
storage       NVMe for models (1.8 TB), SATA SSD for root
OS            Pop!_OS, kernel 7.1.5 with a local 2-line p2pdma patch
runtime       ROCm 7.2.4, rootless podman 4.9.3 (crun)

The PCIe link table is not optional, and it is not lspci. On this platform LnkStacheerfully reports "32 GT/s x16" — Gen5 — for cards on a Gen3 bus. Read pp_dpm_pciefor AMD, and cross-check the upstream root port's AER counters. A card that prints no link state at all is itself the finding:

               link (pp_dpm_pcie)   root port   AER correctable   host read
GPU0 05:00.0   8.0 GT/s x16         03:00.0     0                 13.62 GB/s
GPU1 09:00.0   8.0 GT/s  x8         07:00.0     0                  6.89 GB/s
GPU2 86:00.0   8.0 GT/s x16         84:00.0     0                 13.60 GB/s
GPU3 89:00.0   8.0 GT/s x16         87:00.0     0                 13.64 GB/s

GPU1 runs at x8 deliberately: its IIO is bifurcated so the other eight lanes carry the ConnectX, and a working 25 GbE RoCE link is worth more than 6.7 GB/s on one card. That is a trade, not a fault — but see the correction at the end, because for most of this work the table above did not look like this, and I did not know it.

The symptom pointed at the wrong layer

Tensor-parallel size 2 refused to start. Tensor-parallel size 4 started fine. That asymmetry is what made the bug hard to see, and it has a mundane explanation: the custom all-reduce gates on world_size == 2, so at TP=4 it disables itself and falls back to RCCL, which stages through host memory and never asks for a peer pointer. TP=4 was not healthy. It was avoiding the broken path.

Three plausible theories got refuted before the real one surfaced. The first was a teardown race — the harness slept five seconds after stopping a container instead of waiting for VRAM to actually free. That was a genuine bug and it was fixed, but it was not this bug: TP=2 failed identically on GPUs verified idle. The second was an IPC namespace collision between two server instances. The third was device-index confusion. Both were wrong, and both cost hours. Restarting a service and watching it fail again is not a measurement.

The allow-list

Whether the kernel permits peer-to-peer DMA at all is decided in drivers/pci/p2pdma.c. Two mechanisms sit side by side, and they are not symmetric:

static bool cpu_supports_p2pdma(void)
{
    /* Any AMD CPU whose family ID is Zen or newer supports p2pdma */
    if (c->x86_vendor == X86_VENDOR_AMD && c->x86 >= 0x17)
        return true;
    return false;
}

static const struct pci_p2pdma_whitelist_entry pci_p2pdma_whitelist[] = {
    /* Intel Xeon E7 v3/Xeon E5 v3/Core i7 */
    {PCI_VENDOR_ID_INTEL,   0x2f00, REQ_SAME_HOST_BRIDGE},   /* Haswell-EP  */
    {PCI_VENDOR_ID_INTEL,   0x2f01, REQ_SAME_HOST_BRIDGE},
    /* Intel Skylake-E */
    {PCI_VENDOR_ID_INTEL,   0x2030, 0}, ...
    /* Broadwell-EP (0x6f00/0x6f01): ABSENT */
};

Any AMD part of Zen or newer short-circuits to supported without consulting the table. That single line is the entire reason the same four cards worked on the EPYC. Intel parts must match a device ID, and while Haswell-EP is listed same-host-bridge-only, Broadwell-EP has never been added upstream — despite Broadwell-EP single-root machines having been the standard GPUDirect platform of their era. The box in question reports:

$ lspci -nn -s 00:00.0
00:00.0 Host bridge [0600]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DMI2 [8086:6f00]

The ground-truth probe is cheaper than any of the theories that preceded it. The KFD topology exposes a peer-link list per GPU node, and kfd_add_peer_prop()skips link creation entirely when peer access is denied. Every link pointed at a CPU NUMA node. Not one pointed at another GPU.

The second gate

Adding the two Broadwell-EP entries with REQ_SAME_HOST_BRIDGEand booting the patched kernel removed the chipset refusal for same-socket pairs — and produced no peer links at all. The allow-list is only one of four conditions in amdgpu_device_is_peer_accessible(). The one that still failed was addressability:

$ cat /sys/bus/pci/devices/0000:05:00.0/dma_mask_bits
44
$ head -1 /sys/bus/pci/devices/0000:05:00.0/resource
0x0000380800000000 0x0000380fffffffff        # ~55 TiB, above 2^44 = 16 TiB

The GPU apertures sit far above the 44-bit DMA mask, so the address-mask path cannot pass. The driver accepts an alternative — an IOMMU translating domain — but it accepts only IOMMU_DOMAIN_DMAor its flush-queue variant. The machine was booted with iommu=pt, which is a passthrough identity domain and does not satisfy it. This inverts the usual tuning advice: the widely-recommended iommu=pt is exactly wrong here, and plain intel_iommu=on is what makes peer access legal. The alternative is a firmware fix — moving the MMIO high base below 2⁴⁴ — which this board does not expose.

Settings

hardware      4x AMD Radeon AI PRO R9700 (gfx1201, RDNA4, 32 GB each)
              2x Intel Xeon E5-2620 v4 (16C/32T), 377 GB DDR4-2133
              GPUs split 2/2 across sockets: 05:00.0+09:00.0 | 86:00.0+89:00.0
kernel        7.1.5, locally patched: 2 entries added to pci_p2pdma_whitelist
              cmdline BEFORE: iommu=pt amd_iommu=pt
              cmdline AFTER : intel_iommu=on      (no iommu=pt)
rocm          7.2.4
engine        vLLM 0.27.1 in stilldeadcode/vllm-radiance:0.9.3
model         Qwen3.8-27B, native MXFP4 (quark), 18.04 GiB on disk,
              fp8 MTP head rebuilt from AMD's release
drafter       DFlash2-FP8, method=dflash, num_speculative_tokens=7
serve flags   TP=2, GPU_UTIL=0.95, MAXSEQS=8, MAXLEN=262144, CHUNK=8192,
              kv_cache_dtype=fp8, attention backend R4D
tool          BetterBench, 20 measured passes per category

One flag is load-bearing and worth calling out. GPU_UTIL had to drop from 0.98 to 0.95 after enabling the translating IOMMU: free VRAM at startup became 31.21 GiB against the 31.22 GiB that 0.98 requests, and the engine refuses on free VRAM, not total. It is a memory fraction, not a compute target, so it costs about 0.96 GiB per GPU of KV cache and cannot explain a throughput difference in either direction.

Speed

Both columns are warm floors from 20 measured passes per category on the same harness, same host, same day, differing only in kernel and IOMMU mode. The after column ran with less KV cache (0.95 vs 0.98), so the comparison is conservative against the change being measured.

TP=2, 2x R9700, same socket          before (no P2P)   after (P2P)   delta
prefill  2k depth                      1837.6 tok/s     3481.5       +89%
prefill  8k depth                      1917.9           3668.9       +91%
prefill 16k depth                      1925.7           3672.0       +91%
prefill 32k depth                      1916.2           3617.0       +89%
prefill 64k depth                      1884.0           3491.2       +85%
TTFT p50 @ 8k                          3085.4 ms        1613.0       -48%
decode, combined weighted               148.4 tok/s      178.1       +20%
update p50                               29.0-30.5 ms    25.0-25.8   -15%
aggregate @ concurrency 8               388.8 tok/s      448.3       +15%

Prefill nearly doubling while decode moved 20% is the shape you would predict: prefill moves large activation tensors through the collective every layer, decode moves small ones. The published reference for this stack is 22.7 ms/step on two cards; the gap to it closed from roughly 30% to 10–13%.

Those numbers are not hand-copied. The harness is betterbench, which writes a self-contained HTML report next to every results.json. Both reports are below, unaltered except for the node name. Read the badge row first: it is the run’s own record of what produced it, so the two frames can be checked against each other rather than trusted.

Before — r4d_ar: 0, no P2P collective

After — r4d_ar: 1, kernel 7.1.5-p2p, intel_iommu_on

Both frames carry tp: 2, spec: dflash7, kv: fp8, 20 passes per category and a cold prefix cache; they were taken 89 minutes apart on one machine. Only r4d_ar differs. One honest gap: the earlier run predates the kernel and iommu badges being passed as notes, so its header cannot self-certify the kernel it ran on — that comes from the run timestamps, not from the artifact. The badges exist because of this post; a report that cannot state its own config is the failure mode the whole exercise was about.

Quality

Not measured on this harness, and that is a gap, not a footnote. No perplexity or GSM8K run was taken before and after on this box. The argument for expecting no change is that the edits sit below the numerics — kernel peer-routing policy and an IOMMU domain — and no quantisation, kernel-selection or speculative flag differs between the two columns. That is an argument, not a measurement, and it does not meet the bar for a quality claim. A same-harness WikiText-2 and GSM8K pair is owed before any statement about quality is made.

What the topology then dictated

Peer DMA now works within each socket pair and is still refused across sockets — correctly, because peer TLPs do not route over QPI. That is a statement about peer DMA, not about communication: a GPU can always read and write coherent host memory, so cross-socket transfer is a transport to be built and tuned, not an impossibility. Getting that distinction wrong would write off any model too large for one pair.

For a model that does fit in one pair's 64 GB, the measured answer is unambiguous. Two independent socket-local TP=2 replicas, run simultaneously and measured together, beat one TP=4 instance spanning both sockets by a wide margin:

All three at eight concurrent requests total. Two replicas are +35% over the best single instance and +69% over TP=4. One honest loose end: the socket-1 replica is consistently about 8% slower than the socket-0 one under identical configuration (290.7 vs 316.9 at four concurrent), and that is unexplained.

Two traps that cost hours

The second socket pair was unusable for a completely unrelated reason. The launcher passed the same physical index list to both ROCR_VISIBLE_DEVICESand HIP_VISIBLE_DEVICES. ROCR filters the device set first; HIP then indexes into the filtered set. For GPUs 0,1 that is accidentally correct. For GPUs 2,3 it asks for indices 2 and 3 of a two-device set, and every property lookup fails with a message that reads like missing hardware. ROCR takes the physical list; HIP takes a contiguous one.

The other is a silent no-op. The serving stack sets AITER_ROOT_DIRto a mounted cache, but the library recomputes that value from its own package path and ignores the environment entirely:

# aiter/jit/core.py:80
AITER_ROOT_DIR = os.path.abspath(f"{this_dir}/../../")   # env var never read

So its JIT output landed in the container's writable layer and died with the container, rebuilding on every launch, while the cache directory it was supposedly using sat at 8 KB. A cache that is configured is not a cache that is used — check the byte count, not the setting.

The reproducible check

If GPU peer access is silently absent on an Intel host, this is the whole diagnosis in four commands, and the first one usually ends it:

# 1. which uncore, and is it in the allow-list?
lspci -nn -s 00:00.0        # 8086:2f00 listed | 8086:6f00 NOT listed

# 2. the driver's own verdict
dmesg | grep "not supported by the chipset"

# 3. ground truth: does any GPU link to another GPU, or only to CPU nodes?
for n in /sys/class/kfd/kfd/topology/nodes/*/; do
  grep -H node_to "$n"/p2p_links/*/properties 2>/dev/null
done

# 4. addressability: is the aperture below 2^44 = 0x100000000000 ?
cat /sys/bus/pci/devices/0000:XX:00.0/dma_mask_bits
head -1 /sys/bus/pci/devices/0000:XX:00.0/resource

A firmware investigation ran in parallel to this and produced a clean negative: no BIOS setting can make an uncore forward peer TLPs it refuses, the only firmware-shaped lever is where MMIO lands, and on a whiteboxed board with no recovery path a reflash is not a risk to weigh but an option to remove. The fix was two lines of kernel source and one boot flag. It is worth checking which layer is actually saying no before assuming the hardware is.

Corrections

Three things in the account above were wrong or incomplete, and the measurements that killed them arrived after it was written. They are better content than the original claims.

Two of the four PCIe links were degraded the entire time. GPU1 had fallen to x8 while its root port logged 6 655 BadTLP, and GPU3 had trained at Gen1 — 3.51 GB/s against 13 GB/s for a healthy card — and reported no link state at all. The cause was loose cabling. After re-fastening and a reboot, GPU3 went to 8.0 GT/s x16 at 13.64 GB/s, a +289% improvement on that card and +34% aggregate host-read bandwidth, with every AER counter back to zero. PCIe trains once at link-up and holds for the life of the boot, so fastening the cables changed nothing until the machine was restarted. The AER counters being static — 6 655 unchanged across a 90 s sample — is what proved the damage happened at training time rather than being ongoing.

The consequence for every serving number in this post: each replica held exactly one degraded card, so there was no unaffected control. Before/after deltas measured on identical hardware — the prefill figures — hold their direction. The absolute throughput numbers do not represent this machine repaired, and should be read as provisional. The clearest illustration is the 8% asymmetry noted earlier as unexplained: it was entirely GPU3's Gen1 link, and it inverted after the repair. The socket-1 pair went from 15% slower than socket-0 to 10.5% faster (485.4 vs 439.1 tok/s at eight concurrent), because socket-0 is now the pair carrying the x8 card. Re-measured on repaired links, the healthy pair gives combined decode 186.2 tok/s, prefill 4005–4280 tok/s across 2k–64k, aggregate 488.0 tok/s at eight concurrent, and TTFT p50 of 367.3 ms at 2k — putting prefill within about 7% of the published reference, from roughly 60% below it before the repair.

"Cross-socket is dead" was the wrong sentence — and the first correction to it overshot.Peer TLPs genuinely do not route over QPI, but a GPU can always read and write coherent host memory, so cross-socket communication needs no peer DMA. The first measurement said cross-socket was the fastesttwo-rank pairing; that was an artifact of comparing against the socket pair holding the Gen1 card. Re-measured against a fair control after the repair, the socket crossing costs 0.7–9.3% — negligible, not negative. The structural claim survives and is stronger for being smaller: host reads measure 1.00xwhichever socket homes the memory. QPI was never the bottleneck; the GPU's own PCIe link was.

The obvious fix does not pay, for a structural reason worth knowing. The natural design is hierarchical — reduce inside each socket over peer DMA, cross once, broadcast back. It fails here because peer DMA is roughly half the host-read rate: 5.12–7.31 GB/s between two GPUs on a socket against 13.6 GB/s to host memory. The two GPUs on a socket sit behind different root ports, so peer traffic takes two PCIe hops through the IIO while host staging takes one hop each way at full width. There is no shared switch to short-cut. The hierarchical floor came out at 24.59 µs against a flat collective measured at 27.50 µs — an ~11% edge sitting inside this work's own 20% methodology gap, from a floor that ignores reduce compute, while tripling the rendezvous count that dominates at decode sizes.

And RCCL will not use peer DMA even where it exists. It host-stages every link through /dev/shm, including the same-socket pair, and this is not a tunable: all six NCCL_P2P_LEVELvalues produce an identical SHM path, and NCCL_P2P_DISABLE=1is a no-op — which proves P2P was never in play, because removing SHM falls back to IB at twice the cost rather than selecting peer DMA. HIP can peer here; RCCL simply will not.

The kernel patch buys nothing inside RCCL. RCCL host-stages every link through /dev/shm(isAllDirectP2p 0), including the same-socket pair the patch enabled. The measured gains are real, but they come from the custom two-rank all-reduce that the patch unblocked — not from RCCL suddenly using peer DMA. "I enabled P2P and everything got faster" was too broad a story for what changed.

One more worth recording for anyone on RDNA4: stock RCCL 2.27.7 selects RING/LL and live-locks on gfx1201 — every process at 100% CPU, every GPU at 100%, no error, reproduced 5/5. Production survives only because the serves carry NCCL_PROTO=Simple.

Sources: drivers/pci/p2pdma.c, amdgpu_device.c, ROCm: BAR access limitation, pci.ids.