Cut miren log noise: ~80% of garden's journal is no-op reconcile chatter
While digging through journalctl -u miren on miren-garden (prod runs at debug for now, so this is what we actually see day to day), the signal-to-noise ratio is rough. A clean ~1.9h steady-state window (15:50→17:44 UTC Jun 30, post-MIR-1279-restart) held ~80k lines, about 704 lines/min, which extrapolates to roughly 1M lines/day. Three loops account for ~80% of that volume, and almost all of it is no-op heartbeating that logs every tick whether or not anything changed.
This is the same theme that bit us in MIR-1279: garden was wedged for 14h before a human noticed, and a journal this noisy is exactly where a real signal goes to hide. Worth a cleanup pass.
I pulled the route-lookup ERROR storm out of the numbers below since that was the MIR-1279 incident itself (6,500 before the restart, 2 after), not steady-state noise.
The culprits, ranked by volume in the clean window:
coordinator.sandboxpool"sandbox counts" (DEBUG) — 26,426 lines, 33%. The single highest-volume line in the whole journal. Fires per-pool every reconcile tick even whenactual == ready == desiredand nothing changed. Should only log on a count change.coordinator.deploymentreconcile narration (INFO) — 24,511 lines, 30%. The loop runs ~28×/min and narrates the entire story at INFO on every pass even for a pure no-op:reconciling app→reconciling app version→found pools to check→cleaning up old version pools→checking pool for cleanup→reusing existing pool→pool is being reused, keeping all references. Seven INFO lines per app per minute regardless of whether anything changed (the counts are identical, ~3,159 each, which confirms it's a fixed loop rather than event-driven). These should be DEBUG or gated on actual reconcile work.runner.sandboxchatter (INFO/DEBUG) — 13,393 lines, 17%.considering sandbox create or update(INFO, 8,230) plus DEBUG siblings (network health check passed,sandbox has no exposed TCP ports,checking sandbox for cleanup), again per-sandbox per tick.
Two smaller ones worth folding in:
[WARN] auto-mounting local storage for app with existing data but no disk config— 936 lines. Repeated every reconcile for 6 apps (dan, clusteragent, victoriametrics, tempo, multipass, chitchat), so ~8 WARNs/min forever. Either this should warn-once-per-app, or those apps genuinely need disk config and we should fix that instead.- Go-stdlib noise.
http: TLS handshake error from <internet IP>is just bots/scanners hitting the public IP with junk TLS — silenceable via the http server'sErrorLog. Separately,traces export ... tempo.miren.garden context deadline exceededis showing up too, which may be a real tempo-reachability problem rather than noise.
The win: gate the top-3 loops on state-change (or drop them to DEBUG/TRACE) and we cut journal volume by ~80% with zero loss of real signal, which makes the next MIR-1279 far easier to spot.