Zero-downtime Miren upgrades

Roadmap Summary: Upgrading Miren shouldn't drop your traffic. Ingress moves to its own process so requests keep flowing while the server restarts.

Problem

When miren upgrades, the server process restarts and takes ingress down with it. Since ingress runs in-process with the coordinator, this means:

• In-flight HTTP requests are dropped (no connection draining)
• The HTTP listener goes down entirely during the restart window
• Lease cache (h.apps map) is lost, forcing sandbox re-activation on first request to each app
• OIDC sessions are lost

User applications experience downtime on every miren upgrade, even when the upgrade has nothing to do with ingress or traffic routing.

Goal

User HTTP traffic should continue flowing uninterrupted during miren upgrades, unless the upgrade specifically requires restarting ingress.

This is the core invariant. The design details below are an initial sketch — expect them to evolve.

Initial Design Sketch

Phase 1: Separate ingress process

Extract ingress into its own independently supervised process so the miren server can restart without affecting traffic.

Some things to figure out:

• Process supervision — How is the ingress process managed? Options include a systemd unit, a containerd container, or miren self-management. Each has tradeoffs around complexity and portability.
• Activator interface — Today ingress calls AppActivator in-process. This needs to work over the wire somehow — either as an RPC service, or by rethinking how lease acquisition works when the coordinator is temporarily down during its own restart.
• Entity store access — Ingress currently uses a loopback RPC client to query HttpRoute entities from the coordinator's entity store. Could it keep doing this (with resilience for coordinator restarts), or should it have its own etcd connection?
• OIDC signing key — Needs to persist across restarts regardless of architecture.
• Binary packaging — Separate binary (miren-ingress) vs subcommand (miren ingress) vs shared binary with mode flag. Affects build, deployment, and versioning.

Version-aware restart

Once ingress is a separate process, we need to know when an upgrade actually requires restarting it. If the ingress code hasn't changed, don't bounce it.

Some options to explore:

• Separate binary with its own content hash — upgrade logic compares hashes
• Build-time version stamp derived from the Go dependency graph of ingress packages
• Something else entirely

The goal is: unchanged ingress code = no ingress restart = no traffic interruption.

Phase 2 (future): Graceful ingress restart

For upgrades that do require restarting ingress, implement graceful handoff (socket passing, connection draining, etc.) so that even ingress upgrades are zero-downtime. This is follow-on work — Phase 1 is the priority.

Current Architecture Reference

Key files for context:

• servers/httpingress/httpingress.go — main ingress server, implements http.Handler
• components/coordinate/coordinate.go — where ingress is created (in Coordinator.Start())
• cli/commands/server.go — where ingress is attached to the HTTP listener
• api/ingress/client.go — HttpRoute entity client
• controllers/ingress/ — route lifecycle controllers
• components/activator/ — AppActivator interface that ingress depends on

Existing clean boundaries that may help:

• Ingress already uses an RPC client for entity queries
• AppActivator is a well-defined interface
• Server implements http.Handler — self-contained
• Route config is in etcd and already survives restarts