Submit an issue View all issues Source
MIR-970

Addon deprovisioning blocked when provisioning saga is in-flight

Done Bug public
phinze phinze Opened Apr 3, 2026 Updated Apr 13, 2026

The addon reconciler runs with a single worker, so a long-running provisioning saga (e.g. valkey waiting for sandbox pool to become ready) blocks all other addon reconciliation events, including deprovisioning requests.

Observed sequence:

  1. Server starts, reconciler picks up two addon associations (rabbitmq, valkey) and begins provisioning
  2. Valkey provisioning saga starts waiting for sandbox pool readiness (blocks the worker)
  3. User runs addon destroy which sets status to deprovisioning
  4. The reconciler never processes the deprovisioning event because the worker is occupied
  5. Addons stuck in deprovisioning state indefinitely

Additionally, on server restart the reconciler re-reads old entity revisions and attempts to re-provision addons that were already marked for deprovisioning. The provisioning saga creates new resources (services, pools) that conflict with leftover resources from the previous attempt, compounding the stuck state.

A few possible fixes worth considering: increasing the worker count, making the provisioning saga cancellable when the association status changes, or adding a pre-check in provision() that re-reads the entity to bail out if it's been marked for deprovisioning.