Submit an issue View all issues Source
MIR-1009

Coordinator doesn't detect dead sandboxes on failed runners

Done public
phinze phinze Opened Apr 13, 2026 Updated Jun 29, 2026

When a distributed runner dies (process crash, node failure), the coordinator has no mechanism to detect that the runner's sandboxes are gone. Sandbox entities stay in running status forever because only the runner process updates them. The coordinator keeps routing traffic to the dead sandboxes, causing request timeouts.

Tested on Toys with two runners: stopped runner-1 and killed its sandbox processes. The coordinator kept routing to the dead IPs, resulting in ~15% of requests timing out (2 of 4 sandboxes dead). No rescheduling happened until the runner was manually restarted.

The runner's ready status does fall away correctly (session-scoped attr), but the coordinator doesn't use that signal to stop routing to that runner's sandboxes. The scheduler already checks for ready status when placing new sandboxes, but there's no reconciliation loop that detects "this runner is not ready, its sandboxes should be marked dead."