Submit an issue View all issues Source
MIR-1780

Stale-pool drain that never completes wedges `meet` and hot-loops pool writes every minute

Ready To Merge public
phinze phinze Opened Sep 4, 2026 Updated Sep 4, 2026

Garden has been running this loop roughly every sixty seconds since at least 2026-08-27:

[INFO]  draining stale disk pools before creating new pool │ app: app/meet stale_pools: 1
[INFO]  scaling down stale disk pool │ pool: pool/pool-CZU3R6aQii4yjbGRRJxaz
[INFO]  updating pool │ desired_instances: 0 references: [] num_refs: 0
[INFO]  pool update successful
[ERROR] failed to drain old disk pools, skipping service │ error: "...still has active sandboxes after 1m0s: context deadline exceeded"

Over an eight-day window that single pool accounted for 9,415 of garden's 23,173 updating pool writes, or 41% of every logged pool write on the cluster. Since #1150 landed and removed the competing scale-up/scale-down oscillation, it is now effectively all of them: in a 56-minute window garden logged 57 updating pool against 57 draining stale disk pools, and distinct pools written is 1.

Still looping as of 2026-09-04 17:22 UTC, once per minute with no gaps across 50 consecutive minutes, on main:3caa974 which already contains #1150. It sped up slightly after that fix, 52/hr to 59/hr, because the launcher's worker is no longer blocked cold-starting idle apps, so the retry now cycles close to its 60-second timeout.

Two problems are stacked here, and they want separating.

The first is that the pool never drains. drainStaleDiskPools sets DesiredInstances = 0, then waitForPoolDrained blocks until PoolReadyTimeout, times out, and returns an error that aborts the whole service. Something is keeping that pool's sandboxes alive past a scale-to-zero. MIR-1472 (a terminal disk_lease FAILED with no requeue) is a plausible culprit but I have not confirmed it.

The second is that every attempt rewrites state that is already correct. drainStaleDiskPools sets DesiredInstances = 0 unconditionally, and updatePool calls l.EAC.Replace(ctx, finalAttrs, 0) with no comparison against what is stored, so a pool that has been at zero for a week gets written to zero again every minute. A compare-before-write guard would take garden's pool-write rate to approximately zero without touching the underlying stuck pool.

Worth being explicit about why MIR-1391 will not catch this on its own. That issue's "write only when state changed" mitigation targets ReconcileController, which already does the right thing: AdaptReconcileController returns entity.Diff(meta.Entity, orig) and applyUpdates skips the Patch on an empty diff. The launcher's updatePool bypasses the framework with a direct EAC.Replace. Any direct EAC write inside a reconcile handler sits outside that guard, and this is unlikely to be the only such call site — an audit of them belongs either here or in MIR-1391's scope.

The user-visible part is the quietest and possibly the worst: failed to drain old disk pools, skipping service means meet's service is not being ensured on any pass. That app's deploy path has been wedged for over a week, and the only signal is an ERROR line in the log.

Found while measuring entity churn for MIR-1777, which needed to know how often sandbox pools are written.