Submit an issue View all issues Source
MIR-1481

etcd's HTTP client port bypasses the mTLS that distributedrunners sets up

Done public
phinze phinze Opened Jul 28, 2026 Updated Jul 29, 2026

Enabling distributedrunners calls SetupEtcdTLS and sets --cert-file, --key-file, --client-cert-auth and --trusted-ca-file on etcd, with certs minted for the host's real IPs because runners dial it over the network. That control is bypassable, and not through misconfiguration.

The bypass

In components/etcd/etcd.go, line 444 builds --listen-client-urls with the TLS-aware scheme, but line 445 hardcodes --listen-client-http-urls to http://0.0.0.0:%d, and --client-cert-auth doesn't apply to it:

"--listen-client-urls",      fmt.Sprintf("%s://0.0.0.0:%d", scheme, config.ClientPort),
"--listen-client-http-urls", fmt.Sprintf("http://0.0.0.0:%d", config.HTTPClientPort),
"--listen-peer-urls",        fmt.Sprintf("http://0.0.0.0:%d", config.PeerPort),

Both ports serve the same keyspace. On miren-garden today, from miren-garden-runner-1, with no credentials over plaintext HTTP:

POST http://10.128.0.37:12381/v3/kv/range  ->  {"count":"310941"}

That's the entity store, including env vars marked sensitive, readable and writable. Line 447 hardcodes the peer URL the same way, which is moot at one node and plaintext raft if that ever changes.

Nothing consumes the JSON gateway

The only non-test reference to HTTPClientPort outside config plumbing is the line that passes it to etcd. No client is constructed against it anywhere: the health check (httpingress.go:1028), the bloat/defrag path (components/etcd/maintenance.go), WaitForReady, and grunge all go through the etcd client on the gRPC port.

Live connection counts agree. On garden under real distributed load:

Port Established
:12381 0
:12379 6 (4x 127.0.0.1, 1x 10.128.0.47 runner-1, 1x 10.128.0.46 runner-2)

eu1 is the same: 0 on :12381.

Why it matters for the default flip

The GCP clusters (garden, toys, club) are covered by a default-deny network firewall, so the reachable set there is the VPC. Verified: all of 12379/12381/8428/9428/5000 are blocked from garden's external IP, and allow-miren-web only opens 80/443/8443/8989.

eu1 has no such layer and was serving these to the internet until we firewalled it at the host today (mirendev/infra#93). With distributedrunners becoming the default, the population most affected is self-hosted boxes with nothing in front of them, which is the deployment shape Miren is pitched at.

Suggested fix

Bind --listen-client-http-urls to 127.0.0.1, and have the peer URL honor scheme. Since nothing dials the JSON gateway, this breaks no consumer, and it preserves whatever port separation motivated setting the flag (setting it is what makes --listen-client-urls gRPC-only). Dropping the flag entirely would also close it, moving the JSON gateway back onto 12379 where TLS and --client-cert-auth cover it, but that's the larger behavioral change.

Related

  • mirendev/infra#93 confines these listeners on eu1 at the host level
  • Worth deciding separately whether VictoriaMetrics (8428) and VictoriaLogs (9428) need an auth story before they're runner-reachable; both are handed to runners at Join and neither authenticates.