Make ReconcileController coalesce work and reconcile current state
ReconcileController currently turns watch notifications and periodic anti-entropy scans into queued entity snapshots. Each controller has its own 1,000-item channel. A periodic resync lists the full index and synthesizes an update for every record. When a worker falls behind, pendingEvents preserves every snapshot for a key while the shared channel begins dropping newer work.
This is a correctness problem, not only write amplification. A controller can reconcile stale state after a newer transition has happened. A dropped watch event may not be repaired promptly, and controllers without a periodic resync have no repair path at all. MIR-1753 captures one concrete failure mode: delayed work for a PENDING sandbox can reach the boot path after the sandbox has become terminal.
Change the controller contract so queued work means “this key may be dirty,” not “process this historical snapshot.” Keep at most one pending item per key and read current state when work begins. Anti-entropy should mark keys dirty without flooding the same queue it is intended to repair.
Preserve and complete the mitigations already identified here:
- write only when state changed
- add bounded failure backoff and global admission control
- expose per-controller queue depth, queue age, drops, coalescing, and write velocity
- ensure a dropped watch notification has a reliable repair path
This issue owns the controller-wide fix behind MIR-1706 and the failure mode documented in MIR-1753.