Submit an issue View all issues Source
MIR-1391

Make ReconcileController coalesce work and reconcile current state

Ready To Merge public
phinze phinze Opened Jul 14, 2026 Updated Sep 6, 2026

ReconcileController currently turns watch notifications and periodic anti-entropy scans into queued entity snapshots. Each controller has its own 1,000-item channel. A periodic resync lists the full index and synthesizes an update for every record. When a worker falls behind, pendingEvents preserves every snapshot for a key while the shared channel begins dropping newer work.

This is a correctness problem, not only write amplification. A controller can reconcile stale state after a newer transition has happened. A dropped watch event may not be repaired promptly, and controllers without a periodic resync have no repair path at all. MIR-1753 captures one concrete failure mode: delayed work for a PENDING sandbox can reach the boot path after the sandbox has become terminal.

Change the controller contract so queued work means “this key may be dirty,” not “process this historical snapshot.” Keep at most one pending item per key and read current state when work begins. Anti-entropy should mark keys dirty without flooding the same queue it is intended to repair.

Preserve and complete the mitigations already identified here:

  • write only when state changed
  • add bounded failure backoff and global admission control
  • expose per-controller queue depth, queue age, drops, coalescing, and write velocity
  • ensure a dropped watch notification has a reliable repair path

This issue owns the controller-wide fix behind MIR-1706 and the failure mode documented in MIR-1753.