Submit an issue View all issues Source
MIR-1320

Entity-store index GC leaks stale entries (27% on garden); disk-lease orphan-sweeper loops forever on phantom leases

Done public
phinze phinze Opened Jul 7, 2026 Updated Aug 4, 2026

While baking sagas on garden (MIR-952) I went whitebox on the server logs and found a persistent, self-perpetuating error loop on both garden runners. It turns out to be an entity-store indexing bug, unrelated to sagas.

Symptom. Both miren-garden-runner-1 and -2 have been logging ~48 failed disk-lease "orphan lease sweeps" per hour, continuously since they booted (16:52 on Jul 6). Each sweep tries to patch a disk-lease to RELEASED and the server rejects it:

[WARN] runner.disk-lease orphan lease sweep: patch to RELEASED failed │
  lease: disk-lease/disk-lease-CcdnXcgGgpR7ADh1jQWEd
  error: "... attribute dev.miren.storage/disk_lease.disk_id references a
  non-existent entity: not found: disk/disk-CcdnXcWrvwYDpBcUPAhwJ"

Both runners are stuck on the identical set of 4 phantom leases (CcdnUx…, CcdnXc…, Ccdnar…, Ccdnf8…), which is the tell that this is shared store state, not runner-local cruft.

Root cause: the index outlives the entity. entity get on those lease IDs returns not found, and so do their referenced disks. The entities are gone; only index entries pointing at them survive. miren debug reindex --dry-run on garden confirms the scope: 4,656 of 17,371 index entries (27%) are stale. Entity deletion isn't removing the corresponding index entries, and the garbage has accumulated across kinds (the coordinator also logs ~18/day entity in index but not in store, skipping for the sandbox kind).

Two stacked defects:

  1. Index GC doesn't remove index entries when the backing entity is deleted. That's the 27%.
  2. The disk-lease orphan-sweeper can't make progress on a phantom and retries forever. Worth noting the sandbox consumer already tolerates this exact condition (it logs "skipping" and moves on); the disk-lease sweeper should do the same, pruning a lease whose entity or disk is not-found instead of re-patching every 5 minutes.

Impact: low. No serving impact, no actually-leaked disks (they're already deleted). But it's permanent noise (~2,800 lines/day/runner across the WARN + resync-ERROR pair), it degrades log signal-to-noise during exactly the bake window we care about, and the 27% index bloat keeps growing until the GC bug is fixed.

Remediation. miren debug reindex (no dry-run) rebuilds all indexes and should clear the 4,656 and silence the loop. We're deliberately holding off to preserve evidence for this bug. There are also 2 genuinely-in-store orphan leases (ohfec-preview-pg, on the coordinator) that lease-delete would clean, separate from the phantoms.

Found on garden, all nodes main:9d0af1c. Unrelated to the saga rollout; surfaced by it.