Submit an issue View all issues Source
MIR-2266

Server upgrades always fail their ready check on clusters using ACME DNS-01 certs

Ready To Merge public
phinze phinze Opened Nov 15, 2026 Updated Nov 16, 2026

On a cluster whose ingress gets its certificates through ACME DNS-01, every server upgrade fails its ready check and rolls back, even when the new version is running fine. The ready probe opens a TLS connection to 127.0.0.1:443 without SNI, and the DNS-01 ingress refuses handshakes that carry no SNI. That means automatic upgrades from Miren Cloud can never succeed on these clusters.

What happens

The upgrade operation probes https://127.0.0.1:443/.well-known/miren/health (DefaultHealthURL, pkg/serverlifecycle/prober.go) and sends an empty ServerName. That relied on the autocert ingress, which answers SNI-less handshakes with its self-signed cert. The DNS-01 certificate controller doesn't. It rejects the handshake (http: TLS handshake error from 127.0.0.1:...: no SNI provided), the probe sees remote error: tls: internal error, and after 3 minutes the operation fails.

Observed on a single-node systemd cluster, upgrading v0.17.1 to v0.17.4, with this server config:

[tls]
acme_dns_provider = 'dnsimple'
acme_email = '...'
standard_tls = true

A second, otherwise similar cluster on the default autocert ingress upgraded from v0.17.1 to v0.17.4 in 37 seconds without trouble.

Evidence

With and without SNI, from the server host, against the same server process:

$ curl -sk https://127.0.0.1:443/.well-known/miren/health
TLSv1.3 (IN), TLS alert, internal error (592)

$ curl -sk --resolve tasks.example:443:127.0.0.1 https://tasks.example/.well-known/miren/health
{"status":"healthy", ... "server":{"version":"v0.17.1", ... "ready":true, ...}}

Plain HTTP on port 80 doesn't help either: it answers 302 to HTTPS, and the prober deliberately doesn't follow redirects.

The new version was healthy while the probe failed. In the 3-minute window after v0.17.4 started, the journal shows 80 no SNI provided handshake rejections from 127.0.0.1 and no other real errors.

Rollback "fails" the same way. The auto-rollback to v0.17.1 also reported server not ready after rollback, while every app route was serving normally.

The workaround works but records a failure. miren cluster upgrade --version v0.17.4 --no-auto-rollback leaves v0.17.4 in place with apps healthy. The operation is still recorded as failed, and Cloud shows it as the last operation.

Impact

  • Every upgrade on these clusters needs --no-auto-rollback and a manual health check, which drops the safety net the ready check exists to provide.
  • Cloud-scheduled automatic upgrades can't work on them. Each new build on the policy's channel triggers an upgrade that fails the check and rolls back (two restarts). The scheduler then marks the build held, stuck reminders start after 3 days, and it all repeats with the next build.

The same applies to behind-proxy-https ingress with DNS-01, which uses the same certificate controller.

Fix

The fix is in the ingress, not the probe. The probe checks whichever server is running after the restart, so a fixed server passes even the old probe. That means upgrading onto the fixing release needs no --no-auto-rollback. The one exception is a rollback to an older version, which still can't pass the check. Fixing the probe instead would have meant picking an SNI the ingress has a cert for, which the probe process can't reliably know.