Stale-pool drain that never completes wedges `meet` and hot-loops pool writes every minute
Garden has been running this loop roughly every sixty seconds since at least 2026-08-27:
[INFO] draining stale disk pools before creating new pool │ app: app/meet stale_pools: 1
[INFO] scaling down stale disk pool │ pool: pool/pool-CZU3R6aQii4yjbGRRJxaz
[INFO] updating pool │ desired_instances: 0 references: [] num_refs: 0
[INFO] pool update successful
[ERROR] failed to drain old disk pools, skipping service │ error: "...still has active sandboxes after 1m0s: context deadline exceeded"
Over an eight-day window that single pool accounted for 9,415 of garden's 23,173 updating pool writes, or 41% of every logged pool write on the cluster. Since #1150 landed and removed the competing scale-up/scale-down oscillation, it is now effectively all of them: in a 56-minute window garden logged 57 updating pool against 57 draining stale disk pools, and distinct pools written is 1.
Still looping as of 2026-09-04 17:22 UTC, once per minute with no gaps across 50 consecutive minutes, on main:3caa974 which already contains #1150. It sped up slightly after that fix, 52/hr to 59/hr, because the launcher's worker is no longer blocked cold-starting idle apps, so the retry now cycles close to its 60-second timeout.
Two problems are stacked here, and they want separating.
The first is that the pool never drains. drainStaleDiskPools sets DesiredInstances = 0, then waitForPoolDrained blocks until PoolReadyTimeout, times out, and returns an error that aborts the whole service. Something is keeping that pool's sandboxes alive past a scale-to-zero. MIR-1472 (a terminal disk_lease FAILED with no requeue) is a plausible culprit but I have not confirmed it.
The second is that every attempt rewrites state that is already correct. drainStaleDiskPools sets DesiredInstances = 0 unconditionally, and updatePool calls l.EAC.Replace(ctx, finalAttrs, 0) with no comparison against what is stored, so a pool that has been at zero for a week gets written to zero again every minute. A compare-before-write guard would take garden's pool-write rate to approximately zero without touching the underlying stuck pool.
Worth being explicit about why MIR-1391 will not catch this on its own. That issue's "write only when state changed" mitigation targets ReconcileController, which already does the right thing: AdaptReconcileController returns entity.Diff(meta.Entity, orig) and applyUpdates skips the Patch on an empty diff. The launcher's updatePool bypasses the framework with a direct EAC.Replace. Any direct EAC write inside a reconcile handler sits outside that guard, and this is unlikely to be the only such call site — an audit of them belongs either here or in MIR-1391's scope.
The user-visible part is the quietest and possibly the worst: failed to drain old disk pools, skipping service means meet's service is not being ensured on any pass. That app's deploy path has been wedged for over a week, and the only signal is an ERROR line in the log.
Found while measuring entity churn for MIR-1777, which needed to know how often sandbox pools are written.