Runner telemetry needs a path to VictoriaMetrics/VictoriaLogs that doesn't open them unauthenticated

Prerequisite for running distributed runners on a host without a cloud network firewall. This is a design decision, not a bug.

Where it stands

At Join the coordinator hands each runner a VictoriametricsAddress and a VictorialogsAddress (servers/runner/registration.go:367-372). The runner then builds writers straight at them (cli/commands/runner_start.go:258-279):

metricsWriter = metrics.NewVictoriaMetricsWriter(ctx.Log, cfg.VictoriametricsAddress, 30*time.Second)
deps.LogWriter = observability.NewPersistentLogWriter(cfg.VictorialogsAddress, 30*time.Second)

Neither constructor takes credentials. Both components bind a wildcard address (components/victorialogs/victorialogs.go:222, components/victoriametrics/victoriametrics.go:222, both listenAddr := fmt.Sprintf(":%d", config.HTTPPort)). Confirmed on miren-garden: 8428 and 9428 are reachable from miren-garden-runner-1 and answer without credentials.

So today the only thing protecting runner telemetry is the network boundary. On the GCP clusters that's the default-deny VPC firewall. On eu1 it's the host firewall we landed in mirendev/infra#93, which pins both to loopback.

Why "just add auth" isn't the answer

OSS VictoriaMetrics and VictoriaLogs have no built-in authentication. That's already reflected in our own stack: deployments/victoriametrics/run-caddy.sh fronts the deployed VictoriaMetrics with Caddy basic auth on :8080 and reverse-proxies to victoriametrics.app.miren:8428, leaving the backend open behind a private network. There's no flag to turn on for the embedded pair.

The two shapes

Front them. Put an authenticating proxy in front of the embedded VM/VL, mirroring the Caddy pattern we already use. Smaller change, but it adds a component to the embedded stack, needs credential distribution to runners, and still ends with those ports open to the runner network.

Route through the coordinator. Runners already hold client certs from Join and already speak authenticated RPC to the coordinator on :8443. If telemetry went over that path instead of dialing VM/VL directly, 8428 and 9428 never need to open, and they can stay loopback-only permanently. Bigger change to the writer path, but it removes two ports from the runner-reachable set instead of securing them, and it means one authentication story rather than a second one bolted on.

I lean toward the second, but it's a real tradeoff on effort and on how much we want flowing through the coordinator.

Worth noting these two may not need the same answer as the registry. MIR-1477 establishes that the registry can't retreat to the bridge because remote runners resolve cluster.local:5000 to the coordinator, so authentication is its only lever. Telemetry has no such constraint: nothing requires runners to reach VM/VL directly, which is what makes "route through the coordinator" available here and not there.

Sequencing

The order we settled on is: data-tier services get auth, then the firewall opens, then runners. The three legs are MIR-1481 (etcd), MIR-1477 (registry), and this one (VM/VL). eu1 stays standalone until all three land.