Converging stale-index GC so clusters self-heal instead of staying bloated until a manual reindex

The MIR-1320 fixes (#898, #903) stopped the source of stale index entries — deletes are atomic now, and session-bound index entries expire with their lease. But nothing removes the entries that already leaked, and the only cleanup path is a manual miren debug reindex (the CleanupStale: true path at servers/entityserver/entityserver.go:971) that operators have no reason to know exists.

The automatic boot reindex isn't a substitute. checkAndReindex (components/coordinate/coordinate.go:1575) is gated on the index schema hash, so it only fires when index definitions change in code, not periodically — and even when it runs, it uses CleanupStale: false (coordinate.go:1606), meaning it backfills missing entries but never deletes orphans. So a deployed cluster accumulates stale index entries and holds them indefinitely; healing is gated behind a maintenance op nobody runs.

We measured this live on garden: 3,293 stale entries out of 97,804 collection entries (~3.4%), down from the 27% at MIR-1320's filing but non-zero and self-perpetuating for any collection that doesn't get re-listed. The goal here is that a user who upgrades lands on a version that drains this bloat on its own and keeps it drained — no manual step, no operational surprise.

Design direction (settled after discussion): a bounded, rate-limited periodic background GC that scans collection entries in batches and removes those whose backing entity is authoritatively absent. Keep foreground reads pure — explicitly not an inline write-on-read.

Write-on-read is a legitimate, well-trodden pattern (Redis lazy expiry, Postgres index kill-bits, Dynamo read repair), and the existing skip site at entityserver.go:633 ("entity in index but not in store, skipping") is a tempting seam. But doing the delete on the read path couples read availability to write success, injects latency, and risks a write storm during post-upgrade convergence — all worst exactly when the store is already stressed. At our scale (single-digit clusters, ~3% bloat) a plain background sweep is trivial and boring, and keeps writes on write paths. Read-side detection may at most feed the GC as a cheap signal; it must never be the thing issuing deletes.

Why deletion is safe: creates are atomic — CreateEntity writes the entity key and its collection/index entries in a single txn (pkg/entity/store.go:218-228), so an index entry whose entity key is absent under a linearizable read is genuinely orphaned, with no create-in-flight race. Each delete should still be CAS'd on the index entry's mod-revision so a collection slot legitimately re-created between scan and delete is never clobbered (ABA guard).

Sketch:

• Background worker: bounded batch size + rate limit, idempotent, safe to run continuously; reuse the stale-detection already in the Reindex CleanupStale path (pkg/entity/reindex.go) rather than reinventing it.
• Strictly best-effort: never blocks or fails a foreground op; errors logged, not propagated.
• Metrics for stale-found / stale-removed per sweep so we can watch a cluster converge.
• Behind a feature flag for initial rollout, since it deletes data. Manual miren debug reindex stays as the immediate big hammer.

Adjacent, out of scope but noted:

• The boot reindex writes the new index hash (coordinate.go:1616) even after a deadline-truncated pass, so a partial reindex won't retry. If we ever want to flip that path to CleanupStale: true, it has to become resumable first.
• Open question: is the 3,293 pure legacy, or is something still leaking? The 4 stale sandbox orphans seen over 2h on garden carried newer xids than MIR-1320's original phantom leases — a faint hint of a residual source. A convergent GC would mask an active leak, so we shouldn't let it substitute for confirming the source is actually dead.