Retry-safe range operations¶
Architecture for the retry-safe public range-operation surface and truthful
cleanup outcomes. Decision of record: ADR-063; design guidance:
docs/architecture/public-range-operations-preflight-2086.md (issue #2086).
Retry identity¶
The public retry identity is (deployment_scope, actor_key, action, caller_key):
- deployment_scope is server-owned, resolved by
shared.deployment.resolve_deployment_scope()from the deployment's cloud project (else a configured deployment name)—never a hostname,ENVIRONMENTlabel, workspace UUID, or catalog id. One deployment is one customer boundary backed by one PostgreSQL database (ADR-054), so the scope is recorded on each binding to detect a database restored or cloned into a different deployment. - actor_key is the active actor id resolved via
shared.api.principals.active_actor_user, so a session and a user-owned token for the same actor share identity while each credential still passes its own live checks. - action is
<resource>:<operation>(a launch israes-range:provision). - caller_key is the bounded, caller-supplied key.
Immutable-intent binding¶
shared.operation_intent owns the intent-projection version
(INTENT_PROJECTION_VERSION, distinct from the HTTP, RAES-producer, and
worker-envelope versions) and canonical_intent_digest, which validates a
projection (rejecting non-finite numbers and non-JSON shapes before hashing) and
reuses shared.operation_envelope.canonical_payload_digest for an order-independent
digest.
- The caller intent (the caller-controlled launch selections) is digested for the retry key so a replay recovers without recompiling the server-derived plan even after package/registry/default-configuration changes.
- The compiled intent (the full
OperationInput—compiled plan plus artifact/configuration/delivery/participant-access bindings) is enforced by the engine:engine.launch_intents.enqueue_provisioner_launchnow compares a re-enqueue's composed intent against the stored immutableOperationInputand fails closed on a mismatch, so an internal replay with different bound intent conflicts before effects.
Binding persistence and concurrency¶
engine.models.PublicOperationRetryBinding
(engine_public_operation_retry_binding) is an idempotency/admission index,
not a second execution ledger: it references the existing request/operation and
never owns lifecycle, status, or effects (ADR-043-R3/R4). A UniqueConstraint on
(deployment_scope, actor_key, action, caller_key) lets PostgreSQL arbitrate
concurrent first use.
engine.retry_binding.bind_public_operation recovers an existing binding, else
runs a caller-supplied mint inside the binding transaction and inserts the
binding. When two callers race the same key, the unique insert fails for the
loser with IntegrityError; the loser's whole transaction (including everything
mint reserved) rolls back, and the committed winner is read outside the failed
savepoint and recovered (or conflicts). Same-key contenders therefore converge on
one binding and one set of effects; the real-PostgreSQL proofs live in
tests/engine/test_retry_binding_postgres.py.
The launch orchestration is cms.services.resolve_retry_recovery /
cms.services.bind_first_use_launch; it reaches
the retry-binding primitives only through the engine.services facade (ADR-001)
and is called from mission_control.api.ranges.LaunchRangeView when an
Idempotency-Key header is present.
Reauthorization¶
Replay, status lookup, and cancel reauthorize the current actor on every call through the existing bearer-first authentication, exact scopes, and range ownership. Actor identity is part of the retry key, so a replay can never recover another actor's operation, and a revoked actor cannot authenticate.
Provider inventory/readback evidence¶
Verified terminal cleanup requires an independent inventory/readback of the
owned provider resources, not a logical lifecycle status (ADR-063-R4). After the
RAES destroy delete loop, the provisioner (raes_gcp_inventory.inventory_raes_range_cell)
GETs every resource the plan owns (instances, addresses, routers, firewalls,
subnets, network) on the same enumeration it deleted: NotFound is gone, a returned
resource is a residual, and a provider error is INCOMPLETE (unknown, never an
empty success). The result rides the terminal destroy result payload
(cleanup_inventory, bounded and validated in shared.operation_result_payloads),
and the Engine applier records it as durable RangeCleanupVerification evidence
(scope + observation time) in the same transaction as the DESTROYED transition.
The pinned immutable OperationInput plan makes the destroy upgrade-safe for
resource identity across producer/pack changes; the inventory readback is the
residual safety net that catches anything a drifted configuration missed.
Truthful teardown and retention¶
All three cleanup consumers gate on the latest RangeCleanupVerification being
VERIFIED_ABSENT, never on logical status:
- Verified-terminal reporting.
engine.services.project_range_cleanup_outcomereportsverified_terminalonly withVERIFIED_ABSENTevidence; a logicalDESTROYEDrange with no evidence ispendingwith aprovider_inventory_unconfirmedobligation, andRESIDUALS_FOUND/INCOMPLETEareunknownwith residual obligations. - CTF capacity/linkage release.
ctf.services.range.lifecycledispatches the destroy and marks the rangedestroyingwhile retaining the participant/range/reservation linkage; capacity is released and the linkage cleared only whenctf.signals.sync_ctf_participant_range_statusreceives a DESTROYED transition carryingcleanup_verified=True(computed by the CMS range- event handler from the inventory evidence). Revalidates #1919 (ADR-063-R5). - Retention pruning.
engine.retry_binding.prune_expired_retry_bindings(wired via theprune_retry_bindingsmanagement command) deletes an expired binding only when its operation cleanup is verified absent, so pruning never deletes the sole recovery/residual evidence of an unresolved operation.