Service chains left empty, so every service IP is a black hole

A deployed app with a [addons.miren-postgresql] addon never becomes reachable. The ingress returns HTTP/2 503 Request timeout after exactly 60s, and the coordinator logs a "Creating service" reconcile every 60s, forever, for the same service.

Found from the cloud side by a workbench agent standing up miren server container install --without-cloud in a sandbox and deploying an app to it. The bug is in controllers/service.

The nft state

The verdict map routes correctly and the endpoint chain is correct, but the service chain between them is empty:

map service_ip4s {
    elements = { 10.10.61.35 . tcp . 5432 : goto service_9yATg4xnMR1TMY3tZLndussDiNitDN4UoiqPUaeHdboM }
}

chain endpoint_61s5kbKpKjJvvJmEP8TXbzxTm29dwCBCVyiypqFufrYz {
    ip saddr 10.8.74.2 jump mark-for-masq
    meta l4proto tcp counter packets 0 bytes 0 dnat ip to 10.8.74.2:5432
}

chain service_9yATg4xnMR1TMY3tZLndussDiNitDN4UoiqPUaeHdboM {
}

Traffic reaches the empty chain, matches nothing, and dies there with no DNAT.

Two faults, compounding

The chain body names a counter it does not declare. writeChainBody (controllers/service/service.go) emits counter name "services", but only Init declares that counter. nft answers a rule naming an absent counter with ENOENT and rolls the whole batch back:

apply GC batch: /dev/stdin:8:87-96: Error: Could not process rule: No such file or directory
add rule inet miren service_9yATg4... counter name "services"
                                                   ^^^^^^^^^^

Init runs once at startup (components/runner/runner.go:1030), so any chain body written against kernel state where that never took effect fails.

The chain cache is updated before the batch is applied, which makes it permanent. This is the more serious of the two. setEndpoints records what it wrote into s.chainEndpoints before nft.Run. A rejected batch leaves the cache asserting rules nft never took, so the next pass sees "no change" and skips the rebuild — while addServiceChain in the same transaction, which references no counter, keeps succeeding on its own and creates the chain. That is how the chain ends up existing and empty, and why it never self-heals.

That second fault converts any rejected nft batch into a permanently empty service chain — a transient kernel error, a conflicting concurrent write, a future rule nft dislikes — with nothing logged on the Create path. The 60s GC error was the only reason this was noticed at all, and that was luck: the GC pass is the one caller that bypasses the cache.

Verification

Against real nftables 1.0.9:

• A rule naming an undeclared counter gives exactly the reported ENOENT and rolls back the whole batch.
• The follow-up batch without the body (the cache-poisoned path) then creates the empty chain, reproducing the reported state exactly.
• add counter is idempotent, and re-adding an existing counter preserves its totals — so declaring it in the body costs no traffic visibility.
• Declaring and referencing in one atomic batch is accepted.

Root cause of the missing counters: not established

Worth recording, because the next person will ask.

• The counters had existed. Init creates the maps and the counters in one atomic transaction (one NewTransaction at service.go:176, one Run at :249, nothing in between), and nothing else in the repo constructs either. The reported state still had service_ip4s populated, so that batch committed.
• They went before any rule referenced them. nft refuses to delete a referenced counter with Device or resource busy.
• A wholesale ruleset flush is therefore excluded: it would have taken the maps too.

That leaves something removing the counters specifically, after Init and before the first chain body, without touching the maps or the table. No candidate was found. The sandbox holding the evidence has since been destroyed, so this is now unanswerable.

One caveat on that chain: "counters absent" was never directly observed — nft list counters never returned — so it is inferred from the ENOENT. The inference looks safe (that error was reproduced from exactly that cause, and nft counters are per-table) but it is an inference.

The fix does not depend on the answer, which is the point: a chain body that declares its own counter depends on no prior state at all.

Repro

miren server container install --without-cloud, then deploy any app with [addons.miren-postgresql]. The app deploys and goes healthy — it binds its port — but any request touching the database times out at the ingress, and docker logs miren shows the 60s "Creating service" loop plus the GC error.