Toys runners' disks filled by miren-runner flannel/etcd watch reconnect log spew (~7.7 GB/day) — SSH wedged
Both miren-toys-runner-1 and miren-toys-runner-2 are unreachable. A forensic snapshot + read-only mount of runner-1's boot disk shows it is 100% full (96G used; inodes only 2% — a volume leak, not inode exhaustion).
Root cause
/var/log/syslog (36.7G) + /var/log/syslog.1 (53.7G) = ~90G of miren-runner spewing a flannel subnet etcd-watch reconnect loop:
registry.go:396] etcd watch channel for /miren/sub/flannel/subnets/<subnet> closed with error etcdserver: mvcc: required revision has been compacted, reconnecting...
registry.go:293] registry: watching subnets starting from rev <N>
The watch repeatedly falls behind etcd's compaction horizon and re-establishes in a tight loop, logging a WARN+INFO every few seconds (~7.7 GB/day). Ubuntu's weekly logrotate can't keep pace, so syslog grew unbounded and filled the 100GB disk on 2026-06-27 (rsyslogd then logs write error ... message lost).
Knock-on effect (why SSH is broken)
On a full disk the Google guest agent can't write authorized_keys, so gcloud compute ssh fails with Permission denied (publickey) even though the key reaches instance metadata. That's why the runners are unreachable. The instances are RUNNING but out of the cluster; both are still registered as node entities (10.128.0.41 / 10.128.0.42).
The real bug
The flannel registry etcd watch should recover from compaction without hot-looping (re-list at a current revision, back off) and should not log at full volume on every reconnect. Secondary hardening: rotate syslog by size as well as weekly, and add disk-pressure alerting on runners.
Notes
- The
chronoflowapp crashlooping on this cluster (app list: "crashed (3x, retry 36s)") is a separate issue, not the cause of the disk fill. - Serial console alone is misleading here: its buffer showed stale April content plus an old
Out of memory: Killed process (miren)that did not reflect current state. The runner was actually alive and spewing right up to 2026-06-27.
Impact
Toys has had no healthy distributed runners (at least since 2026-06-27). Blocks runner-hosted-sandbox coverage for MIR-951 (saga validation).
Diagnosis method (for repro)
Snapshot boot disk → create disk from snapshot → attach to a throwaway VM → mount -o ro,norecovery → du. Don't trust the serial console buffer for current state.