GCP range escape validation runbook¶
The escape validation suite proves that the outer boundary of a GCP VM range cell fails closed before the range is trusted for live fire. A GCP range backend is not event ready until this validation passes (ADR-030-R5). Run it after a deploy and before an event.
The suite runs bounded probes from participant context inside the range cell and attempts the escape paths a participant or agent would try. It is read only: it never provisions, destroys, or mutates network state, opens a listener, or edits routes, DNS, or firewalls.
What it validates¶
Each check names the exact boundary it probes, so a failure identifies which outer boundary leaked:
| Boundary code | Expected result | Meaning |
|---|---|---|
cross_range_private_ip |
unreachable | a peer range's private IPs are not reachable |
cross_range_dns |
no route | platform or cross-range names do not resolve to a useful route |
platform_pod_cidr |
unreachable | the platform pod network is not reachable |
platform_service_cidr |
unreachable | the platform service network is not reachable |
platform_node_ip |
unreachable | platform node IPs are not reachable |
platform_portal_private |
unreachable | portal-private endpoints are not reachable |
gke_gdc_api |
unreachable | the GKE or GDC API is not reachable |
metadata_server |
no cross-tenant credential capability | a participant-controlled guest cannot use metadata credentials to list/read platform or another tenant's data |
internet_egress |
per policy | egress matches the configured ADR-017 policy |
management_ingress |
unreachable | a peer range cannot reach this range's management ports |
A reachable metadata server is a failure when a participant-controlled guest can use its credential beyond the narrowly approved per-range capability. A container-only metadata block is defense in depth: when root on the VM host is in the attacker model, validate that host context too. Internet egress is interpreted through the configured policy, not a fixed rule: an approved destination is an expected pass, everything else is an expected fail.
Prerequisites¶
- One or more provisioned, ready GCP ranges.
- A deployment config file (see below). Run the command from the portal management context.
- For a two-or-more-range run, at least one peer range provisioned at the same time. The peer range is the negative target for the cross-range and management ingress checks.
Deployment config file¶
The platform network inventory and the egress policy are deployment facts. Supply them once per deployment in a JSON file. The platform CIDRs come from the Terraform outputs of the environment:
TF_DATA_DIR="<private data directory>" \
terraform -chdir=platform/terraform/gcp/environments/platform output gke_pods_cidr
TF_DATA_DIR="<private data directory>" \
terraform -chdir=platform/terraform/gcp/environments/platform output gke_services_cidr
TF_DATA_DIR="<private data directory>" \
terraform -chdir=platform/terraform/gcp/environments/platform output gke_nodes_cidr
Initialize that data directory against the state bucket and the
shifter/<deployment>/platform-core prefix of the deployment before you read
the outputs.
Example escape-config.json:
{
"platform": {
"pod_cidr": "10.4.0.0/14",
"service_cidr": "10.8.0.0/20",
"node_cidr": "10.128.0.0/20",
"portal_private_endpoints": ["10.128.0.10:5432", "10.128.0.11:6379"],
"gke_gdc_api_endpoint": "10.128.0.2",
"private_dns_names": ["kubernetes.default.svc"]
},
"egress": {
"mode": "deny-all",
"allowed_cidrs": [],
"canaries": ["198.51.100.10"],
"allowed_canaries": []
}
}
The egress.mode value matches the deployment's settings.range_egress policy
(status-quo, deny-all, allowlist, or none). The canaries are operator
owned targets expected to be unreachable; they prove egress is denied. Configure at
least one canary so the internet_egress boundary can be evaluated; without one
that check records a skip and the verdict fails closed. Under allowlist mode,
also set allowed_canaries to operator-owned, known-live targets that the
sanctioned lane should reach; these are expected to be reachable. The
allowed_cidrs list is the declared policy allowlist and is not probed directly,
because a policy range is not proof that any host inside it is live.
Run against one range¶
python manage.py run_range_escape_validation \
--request-id <RANGE_REQUEST_ID> \
--config escape-config.json \
--output escape-report.json
Run against two or more ranges¶
Add one --peer-request-id per peer range. This enables the multi-range gate,
which fails if the peer checks are skipped:
python manage.py run_range_escape_validation \
--request-id <RANGE_REQUEST_ID> \
--peer-request-id <PEER_REQUEST_ID> \
--config escape-config.json \
--output escape-report.json
Adapters¶
The probe runs in participant context through an adapter:
--adapter native(default) runs the probe over the participant SSH channel on a native range VM.--adapter container --container <name>runs the probe inside a scenario participant container on a Docker-host range. The container name is required; there is no default target.
Read the report¶
The report is a closed, versioned JSON document suitable for a CI or operator
gate. The top-level verdict is passed only when every required core check
passes, and, for a multi-range run, the peer checks pass rather than skip. Each
entry in checks carries the boundary_code, status (pass, fail, skip,
or not_applicable), the expected and observed outcomes, and a bounded
diagnostic. Diagnostics never contain credential material.
The command exits non-zero when the verdict is failed and lists the leaked
boundaries, so a CI or operator gate can block a range that leaks.
Soundness and limitations¶
The suite is built so it cannot mistake "I could not test" for "the boundary is closed":
- Every run includes a positive control (the participant reaching a known-live target). If the control does not pass, the probe environment is untrustworthy and the whole run fails closed rather than reporting a false pass.
- A missing tool or a probe that cannot run is recorded as an error and fails the gate; it is never counted as a secure block.
- A silent drop (timeout) is the only secure result for a should-be-unreachable boundary. A connection reset means the network path reached the host, so it fails the boundary even though the port was closed.
- Each network boundary is sampled across several addresses and ports rather than one endpoint, and cross-range DNS is checked against peer-owned names.
- The guest SSH host key from provisioning state is pinned, so an impostor server cannot return a forged all-secure result. A guest with no recorded host key fails the launch.
Known limitation: datagram (UDP) reachability beyond DNS name resolution is not probed. Enforce UDP egress and cross-range UDP isolation with firewall policy and verify it separately; the suite does not currently assert it.
Catch a misconfigured firewall before deploy¶
The static plan-leak checker
(shifter/engine/provisioner/gcp_range_cell_escape_checks.py) inspects a rendered
range-cell plan for a cross-range or over-broad allow rule and reports the exact
leaked boundary without any cloud call. It runs in the provisioner test suite and
catches an intentionally misconfigured cross-range allow rule in a fixture.
Continuous monitoring seam (#2087)¶
The report is also a runtime containment signal, not only a pre-event gate. Each
run_range_escape_validation run hands the versioned, sanitized report to a
containment sink through
shared.range_escape_monitoring.emit_containment_signal. The default
LoggingContainmentSink emits a bounded structured record (contract identity,
range and request attribution, verdict, per-status boundary counts, and the
failed boundary codes) that Cloud Logging ingests without a SIEM. #2087 implements
ContainmentSignalSink to drive continuous monitoring and containment response
and injects it at the call site. The emit is fail-safe: a sink error is logged and
swallowed so a broken or slow monitor never breaks validation, and that logged
failure is the signal #2087 alerts on for collector loss.
Related¶
- Design and boundary model:
docs/architecture/gcp-range-escape-validation-preflight-1347.md - Boundary controls:
docs/architecture/gcp-range-cell-boundary-controls-preflight-1345.md - Egress policy: ADR-017