Reduce log noise in steady-state miren server operation
Context
After upgrading eu1.miren.dev from v0.6.1 to v0.7.0, we pulled 2 hours of journalctl -u miren logs and did a frequency analysis. The server is producing ~86k log lines in 2 hours (~715 lines/min), with 75% at DEBUG and 24% at INFO. Most of it is content-free steady-state chatter that makes it hard to spot anything interesting.
Top offenders
Ordered by volume, these patterns account for 80%+ of all log output:
1. coordinator request authenticated (34,518 lines, 40% of all output)
Every internal RPC call between coordinator and runner logs a DEBUG line confirming auth succeeded. Pure noise in steady state. Should be removed or downgraded to TRACE.
2. coordinator.httpingress using http route (7,879 lines, 9%)
Every proxied HTTP request logs which route it matched. This belongs in access logs or metrics, not structured logs.
3. coordinator.sandboxpool trio (~16k lines combined, 19%)
Every pool reconcile emits three lines: "Processing event", "reconciling pool", "sandbox counts". The first two are content-free. The counts line has some diagnostic value but fires constantly.
4. coordinator.httpingress renewing lease (3,368 lines, 4%)
Lease renewal on active connections. Normal operation, nobody needs to see this.
5. runner.sandbox check-then-skip pair (~4k lines, 5%)
"checking for existing sandbox" immediately followed by "sandbox already exists, skipping create". Two lines to say "nothing to do."
Other notable noise
coordinator.deployment skipping pool for different app(1,816) — logs every pool it ignores, which is backwardsetcd failed to lock file(240, at INFO) — fires twice a minute, might be benign contention but worth investigating separatelyrecorded container stats(668) — stats collection chattercoordinator.sandboxpool healthy sandbox detected, resetting crash counter(1,415) — every health check on every sandbox
Suggested approach
- Remove entirely:
coordinator request authenticated, the "Processing event" lines, "skipping pool for different app" - Downgrade to TRACE or remove:
httpingress using http route,httpingress renewing lease, the sandbox check-then-skip pair,recorded container stats - Consolidate: The sandboxpool trio could be one line with counts included
- Investigate: The
etcd failed to lock filepattern (240 times in 2h at INFO)
WARN/ERROR lines are fine
The warnings and errors in this window are from the upgrade restart (context canceled, process exits) plus some autocert failures for test domains. Nothing to suppress there.