Auto-mounted local storage bypasses the stale-pool drain, orphaning a duplicate pool

Summary

An app using auto-mounted local storage (existing data, no explicit [[disks]] config) can accumulate a second pool for the same app-version on a spec change, because the drain that's supposed to reap the superseded pool is gated on explicit disk config and skips auto-mounted apps. The orphaned pool lingers with a live version reference; on a coordinator restart the activator can bind ingress to it, it crash-loops into cooldown, and — since the whole app is served from that one bound pool — the app goes fully offline while a healthy sibling pool sits idle. No self-heal; recovery needs a redeploy.

Root cause: the mount is in the sandbox spec but not the config

drainStaleDiskPools (controllers/deployment/launcher.go:1533) already does the right thing: on a spec change it finds superseded pools and dereferences + scales them down (ReferencedByVersions = nil; DesiredInstances = 0) so GC can reap them. But it only runs behind a gate:

// launcher.go:250
if serviceHasDisks(spec, svc.Name) {      // serviceHasDisks == len(svc.Disks) > 0
    l.drainStaleDiskPools(...)
}

serviceHasDisks reads spec.Services[].Disks — the config. The "existing data" auto-mount writes somewhere else entirely — the sandbox spec:

// launcher.go:766 (auto-mount)
sbSpec.Volume  = append(sbSpec.Volume,  {Name: "local-data", MountPath: "/miren/data/local"})
appCont.Mount  = append(appCont.Mount,  {Source: "local-data", Destination: "/miren/data/local"})
// ...never touches spec.Services[].Disks

So an auto-mounted app is disk-shaped in the pool spec but diskless in the config. The volume is enough to make the pool spec drift (a pre-existing pool now shows container[0] mounts mismatch, so the launcher stands up a replacement pool), but serviceHasDisks stays false, so the drain never runs and the old pool is never dereferenced.

That dangling reference is what makes it immortal: reaping and cooldown-drain are gated on a pool being unreferenced (controllers/sandboxpool/manager.go:88,109), and cleanupOldVersionPools explicitly keeps any pool referencing the current version (launcher.go:1246). Nothing else ever strips the ref.

Why the orphan turns into an outage

On restart, recoverPools binds (version,service)→pool to the first pool it iterates and serves every lease from only that pool (components/activator/activator.go:1758, AcquireLease:231-236), with no reconciliation against the launcher's canonical pool. If it lands on the stale one, that pool's sandboxes crash on boot, its per-pool cooldown blocks all ingress for the app, and the healthy pool is never consulted. (This is defense-in-depth, not the root cause — see fixes.)

Fix

Register the auto-mount as a real disk in the config (root fix). When the "existing data, no disk config" case fires, add the local disk to the resolved ConfigSpec.Services[].Disks at config-resolution time, rather than patching the sandbox spec at build time. Then serviceHasDisks is true, drainStaleDiskPools runs, superseded pools are reaped, and the special-case at launcher.go:766 can retire — one representation instead of two half-synced ones.

Defense in depth: the activator should bind (version,service) to the canonical pool (spec-matching / has running sandboxes), or scan all pools for the version and reconcile in watchPools — so that if a duplicate ever arises from any cause, ingress can't get stranded on the wrong pool.

Workaround

Adding an explicit [[services.web.disks]] provider="local" mount_path="/miren/data/local" block to the app moves it onto the disk-aware path (serviceHasDisks → true), so the drain runs on future spec changes and the duplicate can't form. Redeploying with that block minted a single clean pool and let GC reap the orphan.

How this was found

Diagnosed from a real outage: an unattended kernel-upgrade reboot took an auto-mounted local-storage app fully offline (100% 408) for ~90 min while a healthy sandbox in the sibling pool passed health checks the entire time. Root-caused against the coordinator journal (boot-0 recovery + the boot-1 origin restart that forked the duplicate) and the runtime source. Full evidence trail available on request.