MIR-1223

Miren app run and sandbox exec fail on garden

Done public

evan Opened Jun 5, 2026 Updated Jun 6, 2026

Refresh runner certificate on start when listen IP changes

Pretty sure it's that the cert being used by the non-primary runners is wrong.

❯ miren sandbox exec -i 5S7 ls /var/run/miren
ERROR: remote error: generic unknown: failed to connect to node 10.128.0.47:8444: error performing http request: CRYPTO_ERROR 0x12a (local): tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, ::1, 10.128.0.45, not 10.128.0.47

I'm pretty certain that 10.128.0.47 is miren-garden-runner-1, but you can see the cert it's using is not for that ip. Additionally, the primary host is 10.128.0.38.

❯ miren sandbox list | grep 5S7
5S7  reviewagent      9hW      web              v5e   10.8.64.3/24   miren-garden-runner-1  running  1h ago   10s ago

Here is the hosts:

❯ multipass gcloud miren-development compute instances list
NAME                   ZONE           MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP     STATUS
dns-forwarder          us-central1-a  e2-micro                    10.128.0.48  34.59.19.161    RUNNING
miren-club             us-central1-a  e2-standard-4               10.128.0.35  34.27.122.56    RUNNING
miren-garden           us-central1-a  e2-standard-4               10.128.0.37  34.122.229.118  RUNNING
miren-garden-runner-1  us-central1-a  e2-standard-2               10.128.0.47                  RUNNING
miren-garden-runner-2  us-central1-a  e2-standard-2               10.128.0.46                  RUNNING
miren-toys             us-central1-a  e2-standard-4               10.128.0.43  104.155.181.29  RUNNING
miren-toys-runner-1    us-central1-a  e2-standard-2               10.128.0.41                  RUNNING
miren-toys-runner-2    us-central1-a  e2-standard-2               10.128.0.42                  RUNNING

Logs on runner-1 indicate that it did, in fact, change ips:

evanphx@miren-garden-runner-1:~$ journalctl -u miren-runner -g 10.128.0.45 | cat
May 19 23:36:03 miren-garden-runner-1.us-central1-a.c.miren-development.internal miren-runner[3462]: I0519 23:36:03.710469    3462 vxlan_network.go:100] Received Subnet Event with VxLan: BackendType: vxlan, PublicIP: 10.128.0.45, PublicIPv6: (nil), BackendData: {"VNI":1,"VtepMAC":"76:a2:cd:b4:1e:fc"}, BackendV6Data: (nil)
May 19 23:37:42 miren-garden-runner-1.us-central1-a.c.miren-development.internal miren-runner[4183]: I0519 23:37:42.841031    4183 vxlan_network.go:100] Received Subnet Event with VxLan: BackendType: vxlan, PublicIP: 10.128.0.45, PublicIPv6: (nil), BackendData: {"VNI":1,"VtepMAC":"76:a2:cd:b4:1e:fc"}, BackendV6Data: (nil)
May 21 03:15:29 miren-garden-runner-1.us-central1-a.c.miren-development.internal miren-runner[20625]: I0521 03:15:29.566628   20625 vxlan_network.go:100] Received Subnet Event with VxLan: BackendType: vxlan, PublicIP: 10.128.0.45, PublicIPv6: (nil), BackendData: {"VNI":1,"VtepMAC":"76:a2:cd:b4:1e:fc"}, BackendV6Data: (nil)
May 21 19:03:14 miren-garden-runner-1.us-central1-a.c.miren-development.internal miren-runner[20625]: I0521 19:03:14.591514   20625 vxlan_network.go:100] Received Subnet Event with VxLan: BackendType: vxlan, PublicIP: 10.128.0.45, PublicIPv6: (nil), BackendData: {"VNI":1,"VtepMAC":"76:a2:cd:b4:1e:fc"}, BackendV6Data: (nil)