RB023: GitHub-less GCP deploy, update, or teardown failed¶
Applicability: GCP deployments that use the GitHub-less deploy path (ADR-070). Operator: platform owner. Verified against the repository: 2026-11-02.
The procedures are in the
GitHub-less GCP deployment guide.
This runbook covers a failed gcp-deploy, gcp-release-scan, gcp-smoke,
gcp-destroy, or gcp-cmek-migrate run, and a bucket key failure in
gcp-foundation. A CI deployment runs the same commands. Its job log shows the
same messages.
Symptom¶
A lifecycle command stops with a status other than 0:
| Exit status | Meaning |
|---|---|
| 1 | A stage failed. The command printed the reason and removed its lock. |
| 2 | The command arguments are not valid. |
| 3 | The deployment is current, but the advisory smoke failed. The command printed The post-deploy smoke failed; the deployment itself completed. |
Before each step, the command prints Running: and the redacted command line.
A failed step prints <tool> <subcommand> failed with exit status <n>. These
messages identify the stage:
| Stage | Messages |
|---|---|
| Inputs | required tools are missing from PATH; deployment inputs must include bigrae.yaml; deployment inputs must include terraform.tfvars or terraform.tfvars.json; range-images.json role ... must be projects/<project>/global/images/[family/]<name>; the deploy tooling revision does not match the release revision; the deploy tooling has local changes; a release manifest is required; settings.release_trust is required to verify release provenance; derived ... identity ... is not a valid service account |
| Lock | the deployment lock is held by <command> run <run id> (started <time>) |
| Infrastructure | settings.operator_bootstrap is required; settings.operator_bootstrap.admin_password_secret_ref is required; the operator password secret has an empty latest version; cannot read the operator password from Secret Manager; GCP operator email must use the ... domain; terraform apply failed with exit status <n> |
| Preconditions | Required GCE range-cell input ... is not set; Range guest image not imported; Range guest image check failed |
| Images | cannot import release image ...; imported image ... resolved to a different digest |
| Scan | exact-release scan failed; the rollout is blocked; failed to store the redacted release verdict |
| Verify | no attestation is stored with ...; no attestation of ... satisfies the release trust policy |
| Rollout | ... belongs to another Helm release; refusing to adopt it; running workloads do not match the approved release; ingress platform-external did not receive a load-balancer address within 10 minutes; certificate platform-managed-cert did not appear for ... within 2 minutes; certificate platform-managed-cert is not Active for ... after 60 minutes; OpenVPN pool servers did not all report healthy within 10 minutes |
| Identity | Legacy identity conversion requires both private facts files; Identity status probe was inconclusive; workloads remain unchanged; A fresh install needs settings.operator_bootstrap; kubectl wait fails on job identity-cutover-status-* or identity-cutover-* |
| Policy | Only some OpenFGA secrets exist; This release has no authorization model; The OpenFGA publisher returned no binding; kubectl wait fails on job openfga-publish-* |
| Catalog | The deploy completes, then warns The in-box catalog may be incomplete or Automatic catalog registration did not complete. The application is active. |
| Teardown | --confirm must equal the deployment name; terraform destroy failed after 6 attempts |
| Bucket keys | Bucket <bucket> default key is not <key>; apply its owner root first; Bucket <bucket>: <n> live objects do not use <key> after the rewrite; storage_cmek_buckets must map each bucket name to its key; Permission denied on Cloud KMS key in a Terraform, upload, or download error |
The warning <bucket>: objects that do not use the bucket key is not a failure.
It counts the noncurrent, soft-deleted, retention-locked, and wildcard-named
objects that keep the Google-managed key.
An OpenFGA publisher error about a missing Authority Key Identifier or Subject Key Identifier means that retained TLS material fails strict verification. Deployment preserves existing secret bundles. Follow the certificate replacement procedure, including live server checks and removal of the old CA after client migration. Keep certificate and hostname verification enabled.
Investigation¶
Record the command, the deployment name, the release revision, and the last
Running: line. Run the checks through the deployer image of the release, as
the guide describes, or from a clean checkout of the release revision.
- Check the tools and the bundle. The preflight renders the Terraform values and makes no change.
Read-only command:
./scripts/bootstrap/deploy.py preflight --cloud gcp --env <environment> --store gs://<bucket>
./scripts/bootstrap/deploy.py gcp-config show --store gs://<bucket>
- For a held lock, read the holder. Find out if that command still runs, for example in a CI run or in the shell of another operator.
Read-only command:
- For a Terraform failure, look for a Terraform state lock. A
default.tflockobject under the prefix is a lock that a stopped apply can leave.
Read-only command:
- For an input failure, check the operator password versions and the range images. Do not read a secret value.
Read-only command:
gcloud secrets versions list <operator-password-secret> --project <project>
gcloud compute images describe <image> --project <range-project>
- For a scan, verify, or running-image failure, read the evidence. The deploy identity can read the release-evidence bucket. Use the newest run ID under the revision.
Read-only command:
gcloud storage ls --impersonate-service-account=<deploy-identity> \
gs://<project>-release-evidence/release-scans/<revision>/
gcloud storage ls --impersonate-service-account=<deploy-identity> \
gs://<project>-release-evidence/deployments/<revision>/
gcloud storage cat --impersonate-service-account=<deploy-identity> \
gs://<project>-release-evidence/release-scans/<revision>/<run-id>/<attempt>/verdict.json
- For a rollout failure, read the cluster state through Connect Gateway as the
deploy identity. Use a separate
KUBECONFIG, and unset both variables when you finish.
Read-only command:
export CLOUDSDK_AUTH_IMPERSONATE_SERVICE_ACCOUNT=<deploy-identity>
export KUBECONFIG=/tmp/<environment>-check
gcloud container fleet memberships get-credentials shifter-<environment>-gke \
--location <region> --project <project>
helm -n shifter-system status shifter
helm -n shifter-system history shifter
kubectl -n shifter-platform get deploy,pods
kubectl -n shifter-platform rollout status deploy/<deployment> --timeout=60s
kubectl -n shifter-platform get events --sort-by=.lastTimestamp
- For a Helm failure that rolled back, read the container logs from Cloud
Logging. The atomic rollback deletes the failed pods, so
kubectl logshas nothing to show.
Read-only command:
gcloud logging read 'resource.type="k8s_container" AND resource.labels.namespace_name="shifter-platform" AND resource.labels.container_name="<container>"' \
--project <project> --freshness=1h --limit 100 --format='value(timestamp,textPayload)'
- For a running-image failure, list the image of each container. Compare each image with the release manifest.
Read-only command:
kubectl -n shifter-platform get pods \
-o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.containerStatuses[*].imageID}{"\n"}{end}'
- For a bucket key failure, read the bucket default key, the key grants, and
the key versions. Each bucket must use its own key, and the Storage service
agent of the project must have
roles/cloudkms.cryptoKeyEncrypterDecrypteron that key.
Read-only command:
gcloud storage buckets describe gs://<bucket> --format='value(default_kms_key)'
gcloud kms keys get-iam-policy <key> --keyring <key-ring> --location <location> --project <project>
gcloud kms keys versions list --key <key> --keyring <key-ring> --location <location> \
--project <project> --format='table(name,state)'
-
For an ingress or certificate failure, compare the ingress address, the certificate status, and the DNS answer.
Read-only command:
Resolution¶
Fix the cause, then run the same command again from the deployer image of the release. Use the guide for each step that changes the deployment.
- Inputs: Use the deployer image of the release (Get and verify the deployer image). Fix a bundle error at its source and publish the bundle again (Author and publish the deployment input bundle). For an identity name that is too long, pass the identity explicitly (Identity options).
- Lock: When the holder still runs, wait for it. When you know that the
holder stopped, run the next command with
--break-lock(Deployment lock). The command removes the lock with no prompt. - Infrastructure: Fix the operator password secret or the operator email
(Create the initial secrets).
A failed Terraform apply does not roll back. Fix the cause in the Terraform
values, publish them, and deploy again
(Update and roll back).
Do not remove a Terraform state lock while an apply can still run. When you
know that the apply stopped, run
terraform force-unlockwith the generation number of the lock object as the lock ID, not the ID in the lock info:gcloud storage objects describe gs://<project>-terraform-state/shifter/<environment>/platform-core/default.tflock --format='value(generation)'. A node pool that staysRUNNING_WITH_ERRORafter a zone stockout fails the apply; follow the stockout entry in the GCP tenant standup gotchas. - Preconditions: Set
settings.range_cells.zoneand import the range images withgcp-images. ForRange guest image check failed, the deploy identity could not read an image; the message names the reason, for example a missingcompute.images.getpermission. For a deliberate platform-first bring-up, deploy with--allow-missing-range-images. - Images: Make the release packages public
(Publish a release),
or deploy from a mirror with
--release-source. - Scan: Deploy a release whose images have no HIGH or CRITICAL finding with
a fix. Read the raw report under
release-scans/. There is no bypass. - Verify: Check
settings.release_trustagainst the repository that published the release. Do not deploy an image that fails its provenance check. - Rollout: The Helm upgrade is atomic, so a failed upgrade rolls back the Helm release. Fix the cause that the events and the rollout status show, and deploy again. For a running-image mismatch, find the owner of the unexpected image before you deploy again.
- Identity: On an empty database the deploy writes the identity facts
from
settings.operator_bootstrap.admin_email; set that block. A database with users needs the reviewed legacy conversion facts (First install). When a cutover Job fails, read itsportalcontainer log in Cloud Logging (resource.labels.pod_name:"identity-cutover"). The Job is gone after the command stops. - Policy: The deploy generates the three OpenFGA secrets
(
shifter-<environment>-openfga-runtime,-admin,-ca) only when none exists. When only some exist, restore the missing ones, or delete all three to regenerate them; then deploy again. A release built before #2679 has no authorization model: deploy a newer release, or pinsettings.openfga.store_idandmodel_id. When the publisher Job fails, read itspublishercontainer log in Cloud Logging (resource.labels.pod_name:"openfga-publish"). - Catalog: The catalog Job acts as the bootstrap administrator that the
cutover recorded, so
settings.operator_bootstrap.admin_emailmust name that account. Otherwise sign in, issue a content credential, and deploy again with--catalog-credential-file <private 0600 file>. - Certificate: Point the DNS A record of the hostname at the ingress
address, with no proxy. Then deploy again without
--first-install(First install). When the certificate does not appear, check thatdeployment.domainnames the hostname. - Smoke (exit status 3): The deployment is current. Find the cause in the
smoke output, then run
gcp-smokeagain (Smoke and exit status). - Bucket keys: For a wrong default key, run the owner command first:
gcp-foundationfor the foundation buckets,gcp-deployfor the platform buckets, or your pack bucket root (Upgrade a deployment to bucket keys). For a missing grant or a disabled key version, follow Repair key access. For live objects that stay unmigrated, find the writer that sets its own key or the identity that can not rewrite objects, fix it, and run the command again. A second run rewrites only the objects that remain. Do not remove a bucket default key to restore access. - Teardown: Pass the exact deployment name to
--confirm. Use the deployer image of the release that the deployment runs. After a failed destroy, rungcp-destroyagain (Tear down).
To roll back, deploy the deployer image of an earlier verified release.
Prevention¶
Deploy only verified releases by digest. Keep the bundle in the config bucket,
not in a repository. Keep the evidence bucket retention lock. Do not disable or
destroy a bucket key version while data uses it. After an upgrade, run
gcp-foundation before gcp-deploy. Run the preflight
before each change. Run one lifecycle command at a time for each deployment.
Escalate when a release fails its provenance check, when the scan finds a HIGH or CRITICAL finding with no fixed release, when the state of a lock holder is unknown, when a Terraform state lock remains after a stopped run, or when a teardown would remove a resource outside the selected deployment, or when a bucket key version is destroyed or scheduled for destruction.