Submit an issue View all issues Source
MIR-1469

TestDiskUndelete flake: lease on a freshly-undeleted disk goes status.failed and never binds

Ready To Merge Bot Friendly runtime Bug public
phinze phinze Opened Jul 24, 2026 Updated Jul 24, 2026

TestDiskUndelete (blackbox distributed-runner suite) flaked on main in run 30105759752 attempt 1; the re-run passed clean, so this is a genuine flake rather than a real regression.

The test walks a disk through create → delete → undelete → lease-the-restored-disk. Everything up through the restore was healthy: the disk came back and debug disk list reported it provisioned (step 8 passed). The lease was then created successfully (exit 0, "Disk lease created successfully"). But the moment it was created the lease went to status.failed, and it stayed failed for the full 60s poll window before the test gave up:

disk_undelete_test.go:109: Waiting for lease to bind...
   → lease-status polled ~30× over 60s, every read: Status: status.failed
disk_undelete_test.go:110: timed out waiting for lease bound (after 1m0s): lease failed
--- FAIL: TestDiskUndelete (62.04s)

We only have the client's view here. Every lease-status line ends with "Status: FAILED - Check entity attributes for error details", and that why lives in the disk-lease controller's server logs on the runner, which aren't captured in the CI artifact. So the symptom is clear but the cause isn't pinned yet.

Leading hypothesis: a race where undelete/restore reports the disk provisioned before its underlying volume img is actually in a leaseable state, so the first bind attempt fails and (unlike the normal provisioning path) never gets retried into a good state. Next step is a local repro via make dev-distributed + make test-blackbox-distributed to catch the controller-side error and confirm.

Worth noting a test-harness sharp edge regardless of root cause: the bind poll loop treats failed as keep-waiting (disk_undelete_test.go:119-121) rather than fatal, so a genuinely-failed lease burns the whole 60s timeout instead of failing fast with the error. Fixing that would make the next occurrence far easier to diagnose.

Cousin issue: MIR-976 (TestChaosConvergence flake — errored disk retains bound lease) sits in the same disk/lease-invariant neighborhood, though it's a different test and the opposite symptom (a lease that stays bound when it shouldn't, vs. ours that never binds).