RB003: Orphaned range resources¶
Applicability: GCP/GCE observation; AWS observation with a limited legacy tool. Operator: platform and range owner. Verified: 2026-10-23.
Symptom¶
Provider resources carry Shifter ownership labels but no active range owns them, or a terminal range still has instances, disks, networks, or secrets.
Investigation¶
- Freeze mutation for the affected range and export a redacted inventory keyed by request ID, operation ID, generation, resource type, and ownership labels.
- Compare provider inventory with Engine ranges, immutable operation inputs and results, launch intents, cleanup verification, and provisioner state. Do not use the CMS projection alone as lifecycle authority.
- On GCP, query resource types separately with purpose-scoped
gcloudlist or describe commands; do not broaden IAM or enable APIs during an incident just to build an aggregate inventory. - On AWS,
shifter-opscan classify EC2 candidates, but its currentreconcile_ranges executepath terminates instances before non-transactional direct SQL updates. Treat it as unqualified for recovery; use observation output only.
Resolution¶
There is no approved provider-neutral orphan mutation. Preserve evidence, quarantine affected networks and credentials through their existing owners, and escalate for a reviewed cleanup plan that names every resource, durable owner, order, rollback boundary, and independent readback. Never delete by name prefix, edit status rows, or remove Terraform state to make inventory appear clean.
Prevention¶
Require ownership labels and operation IDs on every provider effect, make cleanup verification part of lifecycle completion, and run scheduled residual inventory comparisons. A successful destroy must prove both durable convergence and absent provider resources.
Escalate every confirmed orphan; this runbook intentionally has no direct delete command.