Add first-party OTel metric instruments for server operational telemetry
While auditing log noise in MIR-1024 we identified several log lines that were really metrics wearing a trenchcoat. The OTel SDK is already bootstrapped in pkg/rpc/otel.go but we don't currently emit any first-party Counter/Histogram/UpDownCounter instruments from the server itself. (Note: the VictoriaMetrics pipeline under metrics/ is for customer app telemetry, not server operational.)
Natural first instruments
- HTTP ingress request count / latency by app + route_type ("route" vs "default" vs fallback). Replaces the old
using http routedebug log and gives us RED metrics for the proxy. - Controller reconcile count / duration / error rate keyed by entity type, emitted from the generic controller framework in
pkg/controller. Replaces the old INFOProcessing eventlog with actual dashboards. - Sandbox pool gauges: desired / actual / ready per pool. Replaces the
sandbox countsdebug log, which is literally already shaped as a gauge.
Acceptance
- A meter + instruments pattern established somewhere reusable (likely
pkg/rpcor a newpkg/observability). - The three instruments above wired up and verified in a local OTel collector.
- Docs updated with what we expose.