Submit an issue View all issues Source
MIR-1282

Cut miren log noise: ~80% of garden's journal is no-op reconcile chatter

Done public
phinze phinze Opened Jun 30, 2026 Updated Jul 9, 2026

While digging through journalctl -u miren on miren-garden (prod runs at debug for now, so this is what we actually see day to day), the signal-to-noise ratio is rough. A clean ~1.9h steady-state window (15:50→17:44 UTC Jun 30, post-MIR-1279-restart) held ~80k lines, about 704 lines/min, which extrapolates to roughly 1M lines/day. Three loops account for ~80% of that volume, and almost all of it is no-op heartbeating that logs every tick whether or not anything changed.

This is the same theme that bit us in MIR-1279: garden was wedged for 14h before a human noticed, and a journal this noisy is exactly where a real signal goes to hide. Worth a cleanup pass.

I pulled the route-lookup ERROR storm out of the numbers below since that was the MIR-1279 incident itself (6,500 before the restart, 2 after), not steady-state noise.

The culprits, ranked by volume in the clean window:

  1. coordinator.sandboxpool "sandbox counts" (DEBUG) — 26,426 lines, 33%. The single highest-volume line in the whole journal. Fires per-pool every reconcile tick even when actual == ready == desired and nothing changed. Should only log on a count change.
  2. coordinator.deployment reconcile narration (INFO) — 24,511 lines, 30%. The loop runs ~28×/min and narrates the entire story at INFO on every pass even for a pure no-op: reconciling appreconciling app versionfound pools to checkcleaning up old version poolschecking pool for cleanupreusing existing poolpool is being reused, keeping all references. Seven INFO lines per app per minute regardless of whether anything changed (the counts are identical, ~3,159 each, which confirms it's a fixed loop rather than event-driven). These should be DEBUG or gated on actual reconcile work.
  3. runner.sandbox chatter (INFO/DEBUG) — 13,393 lines, 17%. considering sandbox create or update (INFO, 8,230) plus DEBUG siblings (network health check passed, sandbox has no exposed TCP ports, checking sandbox for cleanup), again per-sandbox per tick.

Two smaller ones worth folding in:

  1. [WARN] auto-mounting local storage for app with existing data but no disk config — 936 lines. Repeated every reconcile for 6 apps (dan, clusteragent, victoriametrics, tempo, multipass, chitchat), so ~8 WARNs/min forever. Either this should warn-once-per-app, or those apps genuinely need disk config and we should fix that instead.
  2. Go-stdlib noise. http: TLS handshake error from <internet IP> is just bots/scanners hitting the public IP with junk TLS — silenceable via the http server's ErrorLog. Separately, traces export ... tempo.miren.garden context deadline exceeded is showing up too, which may be a real tempo-reachability problem rather than noise.

The win: gate the top-3 loops on state-change (or drop them to DEBUG/TRACE) and we cut journal volume by ~80% with zero loss of real signal, which makes the next MIR-1279 far easier to spot.