Usage RPC: serve app-level time series from the metrics store

Only getSandbox can return usage as a time series. The app and node views collapse a window into a single number, so there is no way to ask what an app's CPU or memory has been doing over a period.

Answering an app question today means listing that app's sandboxes and reading each one. That only reaches back as far as the sandboxes that are still alive, so any window longer than the current deployment comes back mostly empty. A question about last week, on an app deployed on Tuesday, can only ever cover since Tuesday.

The data is already there

The apparent blocker is that dead sandboxes are expired. What expires is the sandbox record, not the numbers.

• Samples go to VictoriaMetrics and are kept for a month by default (components/victoriametrics/victoriametrics.go:261, defaulted at :94-96).
• Destroying a sandbox calls Metrics.Remove (controllers/sandbox/sandbox.go:3207), which deletes an entry from an in-process map of cgroup managers (controllers/sandbox/metrics.go:122-132). It stops collection and touches nothing in the store. Nothing in the repo calls the deletion API at all.
• The dead sandbox entity is swept about an hour after its last update (controllers/sandbox/sandbox.go:3443-3466, horizon set at components/runner/runner.go:1057-1059).

So the record is gone in an hour and the samples survive for a month, two orders of magnitude apart.

Every sample already carries the app name, along with the service, version, node and kind, stamped in sandboxMetricsIdentity (controllers/sandbox/sandbox.go:1109-1141). The stored label is miren_app. That labelling was added deliberately so an app and its dedicated addons could be summed together (pkg/appspec/appspec.go:124-132).

The conclusion is already written down in the listing code (servers/usage/directory.go:174-181):

a sandbox that was busy an hour ago and has since been replaced is absent from that hour's rows, though its samples are still in the metrics store. Including it would mean reconciling the window against each sandbox's death time, and dead entities are collected eventually, so the answer is to source historical rows from the series rather than from the entity store.

Why this is small

The query builder already takes the grouping label as a parameter: cpuCoresQuery(groupBy, selector, window, aggregate) at servers/usage/query.go:88. The node view already uses that to group by something other than a sandbox (servers/usage/nodes.go:280). An app series is the same query with miren_app in place of miren_sandbox. No new collection, no new indexing, no retention change.

What is missing:

• A labelApp = "miren_app" constant. servers/usage/query.go:30-32 defines only sandbox, node and runner.
• A series result on the app operations in api/usage/rpc.yml. AppUsage has no series field, and getSandbox is the only operation returning a UsageSeries.
• An appSeries builder modelled on sandboxSeries (servers/usage/sandbox.go:233), but keying output series by the grouping label instead of flattening them. The sandbox path flattens every returned series into one list of points (sandbox.go:270-291), which is only safe because its selector pins a single sandbox. Group by app and several series come back and would be concatenated into one undifferentiated run. instantByLabel (servers/usage/usage.go:284-297) already does the keying correctly for the aggregate path.

A stale comment to fix while in there. apps.go:233-238 says the services-versus-addons split has to be made per sandbox "because the metric labels do not carry that distinction". They do now: miren.kind is stamped at controllers/sandbox/sandbox.go:1138 from the same function the entity path uses (servers/usage/directory.go:243), and api/compute/compute.go:38-41 says so outright. sum by (miren_app, miren_kind) recovers the split from the store with no entity join, which removes the last reason a historical query needs the entity store.

Things to decide

• A metrics-sourced listing will surface apps that no longer exist, for a month. Arguably right for a historical question, but it is a behaviour change from the entity-driven listing.
• sandbox_count, service_count and addon_count (apps.go:219-221) have no equivalent in the store. Counting distinct miren_sandbox counts reporting sandboxes, which is not the same as scheduled ones.
• app_id is not on samples, only the app name. entity carries the app id but only for service sandboxes, not addons.
• Cardinality is unmeasured. A new series is minted per collector process (metrics/cpu_usage.go:42) and miren_version changes on every deploy, so a wide grouping over a month may be expensive. Worth measuring on a real cluster before committing to a default window.
• UsageWindow.step_seconds is documented (api/usage/rpc.yml:47-50) but never set — window.encode() does not populate it. A charting client cannot currently learn what resolution it was given. Worth fixing alongside.

One thing that is not a blocker but is worth knowing

Unlike miren.sandbox, miren.node, miren.version and miren.kind, the app label is not re-stamped by the controller. appspec.Build strips a caller-supplied one (pkg/appspec/appspec.go:136-144), but a hand-rolled spec could still claim another app's name. Irrelevant for a dashboard. It would matter if these numbers ever fed billing.

Why now

Miren Cloud's workbench agent has a tool that reads usage over time and draws it (MIR-1828). It works per sandbox and has to tell people it can only cover the period the current sandboxes have been alive. App-level series would make the obvious question answerable properly. MIR-1606 wants point-in-time app resource tiles and would be served by the same query.