Tracing four fleet failures to attribution errors

Between 2026-08-31 and 2026-09-01, four separate failures hit the GPU fleet: a video render died allocating VRAM on a card that looked free, a fleet doctor named two innocent processes as VRAM squatters, a KV-cache capacity target copied from an external report could not be reproduced, and a health check stayed green behind a process that was not the service at all. None of the four was a reasoning failure. Each was an attribution error — an index, an owner label, a bare number, and a green HTTP response, each trusted as an identity. The corrections were mechanical: for future pins, use a persistent device UUID and assert what the process itself reports; resolve an allocation through the device file to argv, carry the artifact and config beside every number, and demand an identity field plus a real completion instead of a status code.

An attribution error fails quietly: the checks ran, the tools answered, and each answer described the wrong object. In every case below the fix was not deeper analysis but a different key — the identifier or semantic payload the question actually needed. The evidence is Horde commits and lane records from 2026-09-01, plus one external artifact reported 2026-08-31.

All Horde commits in the caption are dated 2026-09-01 unless noted. The 6.50x figure compares the fleet checkpoint (125,870,631,446 bytes, du -sb, 2026-09-01) with the external checkpoint’s model.safetensors (19,373,796,656 bytes, reported 2026-08-31).

Case 1: an index is not an identity

On 2026-09-01, one host ran an NVIDIA GeForce RTX 4090 beside an RTX 3090, with two ComfyUI (a node-based image and video generation server) CUDA sidecars. At the time of the incident both were launched withCUDA_VISIBLE_DEVICESunset and without a device flag, so each opened the first device its own runtime enumerated: each process’s owncuda:0. Per the CUDA programming guide’s environment-variables appendix,CUDA_VISIBLE_DEVICEScontrols device visibility and enumeration order, and accepts UUID strings as identifiers, whileCUDA_DEVICE_ORDERcan switch enumeration to “PCI_BUS_ID” — enumeration is per-process configuration, not a fleet-wide name.

Measured 2026-09-01 (Horde commite7f22cbf):nvidia-smicalled the RTX 3090 index 0 and the RTX 4090 index 1, while the target ComfyUI’s own/system_statsendpoint called the RTX 4090 cuda:0 and the RTX 3090 cuda:1. The same index named two different cards depending on who answered. The target reported 0.4/23.5 GiB free on its cuda:0 RTX 4090 — an undeclared 22.9 GB process held the card — while the RTX 3090 sat at 23.3/23.6 GiB free with 4 MiB used. A MiniMax H3 reference-to-video job then failed in its image-encoder stage (vae.encode) withVRAM grow failed: 56625152 bytes.

The installed correction usedCUDA_DEVICE_ORDER=PCI_BUS_ID, masked one PCI-ordered ordinal per sidecar, and failed closed unless the process reported the expected device name. After installation the two instances reported: instance A, cuda:0 “NVIDIA GeForce RTX 4090”, 0.7/23.5 GiB free; instance B, cuda:0 “NVIDIA GeForce RTX 3090”, 22.7/23.6 GiB free. This is a guarded location map, not a persistent identity. Decision: a new launcher should pin by device UUID and still ask the process which device it actually opened; a legacy ordinal map must assert its result and fail closed.

Case 2: an allocation is not process ownership

An AMD device set carries four Radeon AI PRO R9700 cards running a Qwen3.8 Flash-Next serve — TP=4, tensor parallelism splitting the model across all four cards, max context 524288 — alongside three ComfyUI instances. On 2026-09-01 (Horde commitb58459a69) the fleet doctor attributed 2,009 MiB of VRAM on one card to two of those ComfyUI processes and flagged them for eviction. Stopping both freed byte-identical zero VRAM — the allocation had no connection to the accused processes.

fuser -v /dev/kfd— asking the card’s device file who holds it open — named a third ComfyUI process, running under systemd (the system service manager). Its command line declared--reserve-vram 2 --disable-dynamic-vram --disable-smart-memory: the 2,009 MiB was a declared reservation, not a leak and not a squatter. The reservation was then published as scheduler input in Horde commite50725507(2026-09-01). Decision: never recommend eviction from anowner: unknownlabel. Resolve the holder from the device file to PID, then to argv and cgroup (the kernel’s process containment group), and publish declared reservations so the scheduler plans around them.

Case 3: a number without its artifact and config is not a result

An external report (codeberg.org/ggz14/radiance-vllm-mxfp4, reported 2026-08-31, recorded the same day in the fleet’s lane notes) described approximately 920,000 KV-cache tokens on two Radeon AI PRO R9700 cards. Its configuration: TP=2, MAXLEN=262144, GPU_UTIL=0.98, an FP8 KV cache with W4A8 quantization enabled, and the checkpoint Qwen3.8-27B-MXFP4-mtpfp8-r2, whose model.safetensors is 19,373,796,656 bytes (18.0 GiB). The KV cache is the per-token attention state a serve holds; its capacity bounds how much context fits. This figure is external and was not reproduced on this fleet.

The fleet’s configured checkpoint is a materially different artifact: Qwen3.8-Flash-Next-MXFP4-FP8, measured at 125,870,631,446 bytes (117.2 GiB) by read-only du -sb on 2026-09-01 — 131 model shards plus a KV scale file. The serve is pinned to the imagedocker.io/tcclaviger/vllm@sha256:1465c4571b6b24a0ad91ede614f91bccb938158e10a86fff7cbd56b98543e713with TP=4, four multi-token-prediction speculative draft tokens, max context 524288, and a 51 GB PLE CPU offload (Horde commitb58459a69, 2026-09-01). The byte ratio between the two checkpoints is 125,870,631,446 / 19,373,796,656 = 6.50x, and the configurations differ alongside the bytes. Weight bytes alone do not determine KV capacity: vLLM’s kv_cache_utils documentation describes available memory, max model length, and gpu_memory_utilization as the jointly constraining inputs.

Measured fleet bounds on 2026-09-01, same pinned image, checkpoint, and config: gpu_memory_utilization=0.95 refused startup with 29.42 GiB free against 30.27 GiB requested; 0.90 passed the first check but left 3.87 GiB of KV cache against the 3.94 GiB needed for max context 524288; 0.92 fit at full 524288 context alongside the declared 2,009 MiB reservation from case 2, leaving 2,610 MiB by the launcher’s own sizing calculation.

Decision: a KV-cache capacity number is keyed by checkpoint bytes and revision, runtime digest, topology, quantization, context, and memory flags. The external ~920,000-token figure was keyed to a 19,373,796,656-byte checkpoint at TP=2 (reported 2026-08-31) and cannot be compared with a 125,870,631,446-byte checkpoint at TP=4 (du -sb, 2026-09-01). A target copied without its key is not a result.

Case 4: a green response is not service identity

A supervisor’s worker health check stayed green for months on a configured port that a hand-started process — not the configured worker — had been occupying (Horde commitea3cf2a02, 2026-09-01). A generic 200 response proved only that some process answered the port. The correction requires the health body to report the configured runtime root — an identity field the supervisor owns — so a process that cannot name its runtime root fails the check regardless of status code.

The same error class appeared on an NPU host. Hardware: Strix Halo, with its XDNA2 neural accelerator. The exact serve binary, SHA-2567b763b803880f0dc69420f12ec85dcc9e824b93c8ab884cee7ac4eef1a6782cfmeasured 2026-09-01, was launched asflm serve qwen3:1.7b --host 127.0.0.1with FLM_MODEL_PATH set to the configured model root, the XRT library path set, and memlock unlimited (Horde commit1dbb237a6, 2026-09-01; role flm_npu_serve, play npu-serve-verify). This serve implements no /health route: GET /health and GET / both return 404 with Content-Type: multipart/form-data, while GET /v1/models returns 200. A model list alone is still insufficient — a serve pointed at the wrong model root lists all 35 catalogue entries and fails only when a completion is actually requested. The acceptance path therefore checks supervision, GET /v1/models, and one real completion.

The 2026-08-31/2026-09-01 acceptance record proves that one real completion occurred, but it does not bind the observed rate fields to the later binary digest and managed launch environment. Warm-up, concurrency, saturation, and repetition count were also absent. Decision: retain the completion as liveness evidence and omit the unbound rates from this post.

A third variant hid in storage telemetry. Unprivilegedsmartctl --json --all— SMART is the drive self-test dataset — exited cleanly and returned valid JSON, but the envelope contained no temperature, no SMART status, no power-on time, no attributes, and no NVMe health log (fleet collector resource_slices.py, live evidence HORDE-130, 2026-09-01). The collector now treats that semantic emptiness as unreadable and retries through the narrowly authorized read-only path instead of reporting a parseable-but-empty success. Decision: a probe proves exactly the predicate it checks. Service identity needs an identity field owned by the supervisor; route health needs a route the binary implements plus a semantic payload; storage health needs health fields, not a parseable envelope.

The four checks, as commands

Each pattern below is the correction reduced to loopback commands. Placeholders are environment variables; no command needs a hostname beyond 127.0.0.1.

# 1. after configuring a UUID pin, assert the process report
: "${GPU_UUID:?set GPU_UUID}" "${COMFY_URL:?set COMFY_URL}"
nvidia-smi --query-gpu=index,uuid,pci.bus_id,name --format=csv
expected_name="$(nvidia-smi --id="$GPU_UUID" --query-gpu=name --format=csv,noheader)"
actual_name="$(curl -fsS "$COMFY_URL/system_stats" | jq -er '
  .devices | if length == 1 then .[0].name else error("expected one device") end')"
case "$actual_name" in
  *"$expected_name"*) printf 'matched %s\n' "$expected_name" ;;
  *) printf 'mismatch: expected %s, got %s\n' "$expected_name" "$actual_name" >&2; exit 1 ;;
esac

# 2. allocation -> holder: ask the device file, then argv and cgroup
for pid in $(fuser /dev/kfd 2>/dev/null); do
  printf '%s ' "$pid"; tr '\0' ' ' < "/proc/$pid/cmdline"; echo
  cat "/proc/$pid/cgroup"
done

# 3. number -> key: record bytes, digest, and flags together
: "${CHECKPOINT_DIR:?set CHECKPOINT_DIR}" "${SERVE_CONTAINER:?set SERVE_CONTAINER}"
: "${CTX:?set CTX}" "${TP:?set TP}" "${UTIL:?set UTIL}"
du -sb "$CHECKPOINT_DIR"
docker inspect --format '{{.Image}}' "$SERVE_CONTAINER"
printf 'ctx=%s tp=%s gpu_mem_util=%s date=%s\n' \
  "$CTX" "$TP" "$UTIL" "$(date -I)"

# 4a. managed worker -> owned identity field
: "${WORKER_URL:?set WORKER_URL}" "${EXPECTED_ROOT:?set EXPECTED_ROOT}"
curl -fsS "$WORKER_URL/health" \
  | jq -e --arg root "$EXPECTED_ROOT" '.root == $root'

# 4b. NPU API -> implemented route plus a real completion; no /health
: "${NPU_URL:?set NPU_URL}" "${MODEL:?set MODEL}"
curl -fsS "$NPU_URL/v1/models" | jq -e '.data | length > 0'
curl -fsS "$NPU_URL/v1/chat/completions" \
  -H 'content-type: application/json' \
  -d "{\"model\":\"$MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"ping\"}]}" \
  | jq -e '.choices[0].message.content | length > 0'
# storage: parseable JSON is not health — require health fields
: "${DISK:?set DISK}"
smartctl --json --all "$DISK" | jq -e '
  has("temperature") or has("smart_status") or has("power_on_time") or
  has("ata_smart_attributes") or has("nvme_smart_health_information_log")'

Settings

Speed

NPU host — completion/liveness evidence, 2026-08-31/2026-09-01
result                  one real completion
throughput / TTFT       not publishable from the recorded artifact
reason                  no receipt binds rate fields to digest + launch config

GPU hosts — allocation and startup evidence only; no throughput result.

One diagnostic response was retained as liveness evidence. Warm-up, concurrency, saturation, and repetition count were not recorded, and no receipt binds its rate fields to the later digest and managed launch config, so no throughput number is published. The GPU incidents produced allocation and startup evidence — VRAM free/used readings, startup refusals, KV-cache sizing — and no throughput measurement.

Quality

Not measured. No benchmark name, sample count, score, or same-harness baseline exists for this incident window, so this post makes no model-quality claim.

Primary sources for the mechanisms: the NVIDIA CUDA programming guide’s environment-variables appendix and vLLM’s kv_cache_utils documentation, linked above. Full evidence lives in the Horde commits and lane records named throughout; this post is the summary, and those artifacts are the evidence.