Disk volume created on distributed runner node (violates "primary only" invariant)

Invariant in the current state of distributed runners: disks should only ever live on the primary node. On toys today, something created a disk_volume on one of the runner nodes, which has the knock-on effect of permanently stalling addon provisioning.

Repro: deploying rust-chat (MIR-1022 booth work) with [addons.miren-valkey] declared in app.toml. Four consecutive rust-chat-vk-* sandboxes died at "creating disk lease," the addon never came up, and the app sandbox never spawned. Site went to 500.

What the logs show on miren-toys around 2026-04-17 22:30–22:35 UTC for disk disk-CaA8mzapV8JfGhJ5oVTza:

runner.sandbox created disk │ disk: disk-CaA8mzapV8JfGhJ5oVTza status: provisioning
runner.disk creating disk_volume entity │ volume: disk-vol-CaA8mzgGqJmHwjnNFoFWV node_id: miren
runner.disk disk_volume belongs to another node, skipping │
  volume: disk-vol-CaA8mzg2awDYVvfRJMt4d volume_node: node/263c9a60-8957-4085-b8ec-9c4a30c91a0d my_node: node/miren
runner.disk-volume-watch disk_volume changed │ volume: disk-vol-CaA8mzgGqJmHwjnNFoFWV actual_state: dv_ready
runner.disk Processing disk update │ status: status.provisioning
runner.disk-lease Processing lease update │ status: status.failed

Two disk_volume entities exist for one disk:

• disk-vol-CaA8mzgGqJmHwjnNFoFWV on node/miren (primary, correct) — reaches dv_ready
• disk-vol-CaA8mzg2awDYVvfRJMt4d on node/263c9a60-8957-4085-b8ec-9c4a30c91a0d (miren-toys-runner-2) — shouldn't exist

The parent disk never leaves status.provisioning, because the primary's reconciler (correctly) refuses to touch the foreign volume and (less correctly) doesn't finalize the disk state when its own volume is ready. Leases go immediately to status.failed, and after four sandbox retries we give up.

The real bug is the foreign volume getting created at all. Most likely suspect: runner-side disk controllers reacting to disk-entity watch events and creating a local disk_volume even though they shouldn't. I didn't see the creation log for the foreign volume in the window I grepped — worth tracing which component emits it. Secondary fix idea: the primary-side reconciler could at least recognize "my volume is ready, the disk is mine, transition the disk to ready regardless of the junk volume elsewhere" so stuck-state survives an orphan.

Evidence IDs for log grep: sandboxes rust-chat-vk-CaA8mzDW6BQU2e8rFuWkk, rust-chat-vk-CaA8p5N7Zm9jouFTCEfPz, rust-chat-vk-CaA8rLsKwYMHcMcvJua5e, rust-chat-vk-CaA8v5XwwNMvi4X27KHHf.

Collateral (flag, don't scope in): the deploy itself reported success ("Updated version err deployed, all traffic moved") while the addon sat stuck and the app served 500s. Upstream probably shouldn't claim "traffic moved" when no healthy backend exists. Separate ticket if it's worth one.