Addon deprovisioning blocked when provisioning saga is in-flight
The addon reconciler runs with a single worker, so a long-running provisioning saga (e.g. valkey waiting for sandbox pool to become ready) blocks all other addon reconciliation events, including deprovisioning requests.
Observed sequence:
- Server starts, reconciler picks up two addon associations (rabbitmq, valkey) and begins provisioning
- Valkey provisioning saga starts waiting for sandbox pool readiness (blocks the worker)
- User runs
addon destroywhich sets status todeprovisioning - The reconciler never processes the deprovisioning event because the worker is occupied
- Addons stuck in
deprovisioningstate indefinitely
Additionally, on server restart the reconciler re-reads old entity revisions and attempts to re-provision addons that were already marked for deprovisioning. The provisioning saga creates new resources (services, pools) that conflict with leftover resources from the previous attempt, compounding the stuck state.
A few possible fixes worth considering: increasing the worker count, making the provisioning saga cancellable when the association status changes, or adding a pre-check in provision() that re-reads the entity to bail out if it's been marked for deprovisioning.