cluster-wide outage on `club`: `/run` tmpfs inode exhaustion from leaked containerd FIFOs

What happened

On 2026-05-08 around 14:30 UTC, freeq-site (and ~22 other apps) started flapping on the club cluster. New sandboxes went pending → dead within ~130ms with no application output. From the runner journal:

runc create failed: mkdir /run/containerd/runc/miren/sandbox.<id>_pause: no space left on device

But disk was 17% full and /run had 1.8M of 3.2G used. The actual constraint was inodes:

tmpfs  819200 inodes  819199 used  100% /run

Root cause

/run/containerd/fifo had 272,484 directories. Live containers per /run/containerd/runc/miren/: 40. Each FIFO dir contained two named pipes (stdout/stderr) for some past container that had been GC'd from containerd state but whose FIFO dir was never removed. The leak filled the tmpfs inode table, after which every new shim creation failed with ENOSPC and produced more half-stuck state. Self-reinforcing.

Resolution

Surgical cleanup: matched FIFO dir contents against the live container ID set, deleted only orphans. 272,419 dirs removed in 18.7s, no live containers affected. /run inodes went 100% → 0.2%. miren app restart -C club -a freeq-site kicked the pool off its crash cooldown and freeq.at came back up.

Forensic sample of 500 orphan container IDs preserved at /var/lib/miren/orphan_sample.txt on miren-club.

Follow-up threads

This issue is meant to anchor a few separate threads worth tracking individually:

• Bump /run tmpfs nr_inodes on hosts. Default 800k is too tight for our churn rate. While we're in there, add a df -i /run alert at ~80% so this pages before the cluster eats itself.
• Find and fix the containerd FIFO cleanup leak. 237,305 distinct orphan container IDs across 272,419 orphan dirs (~1.15 dirs per ID) suggests a path in container teardown that drops cleanup, not a one-off.
• Pool crash cooldown ratcheting got nasty. Pools hit consecutive_crashes=7 with 10-minute cooldowns. After the underlying issue was fixed, recovery wasn't automatic. Worth thinking about whether the cooldown should detect cluster-wide ENOSPC (vs. per-app crashes) or whether app restart shouldn't be needed after the host recovers.
• Cosmetic but real log spam noticed during investigation: error looking up default route: conflict in entity: more than one entity found (two entities marked default route somewhere) and a stuck disk_lease-CWEjBoCiuPQHgEgxPzQZW referencing a nonexistent pterodon sandbox.