Operations runbooks¶
These runbooks are the operational handoff for the current BigRAE control plane. GCP with GCE range cells is the primary supported release path. AWS procedures are called out only where the repository and a deployed AWS tenant provide an equivalent surface. GDC VM Runtime remains an operator-validation mode and is not a live-fire containment boundary.
The procedures were checked against the repository and deployed service types on 2026-10-23. Resolve names from the selected deployment's Terraform outputs and protected environment; never copy names from an example or another tenant.
Shared operating contract¶
Set non-secret identifiers before using command examples:
export PROJECT_ID="<gcp-project-id>"
export REGION="<gcp-region>"
export CLUSTER="<gke-cluster>"
export NAMESPACE="<platform-namespace>"
export JOBS_NAMESPACE="<jobs-namespace>"
export REQUEST_ID="<request-uuid>"
export OPERATION_ID="<operation-uuid>"
export SINCE="2h"
For AWS-only branches, use the approved profile and set AWS_REGION and
AWS_CLUSTER. Confirm the active cloud identity, selected project/account,
cluster context, namespace, incident window, and operator authorization before
investigation. Production is never an implicit default.
Investigation is read-only. Keep request and operation IDs in evidence, but redact user data, tenant identifiers, provider payloads, signed URLs, database connection strings, and credential material. Never print Kubernetes Secrets, Secret Manager values, process environments, or complete Terraform outputs.
Use the application service boundary for range, account, upload, and credential
changes. Use the protected deployment workflow for infrastructure, image,
migration, or workload changes. A GCP deployment without GitHub uses the
release-driven gcp-deploy command instead (see the
GitHub-less GCP deployment guide).
Direct SQL, hand-built events, hand-run production Terraform or Helm, arbitrary
kubectl exec, and deleting cloud resources by name are not recovery paths.
Escalate when the safe mutation surface is absent, ownership is ambiguous, cleanup cannot be proven, a security boundary may have failed, or recovery would change infrastructure outside the selected deployment. The incident commander owns service restoration; the security lead owns containment and credential revocation; the platform owner owns deployment and Terraform; the data owner approves restore/cutover; the range owner confirms lifecycle and residual inventory.
Treat application quotas, cloud service quotas, event-capacity profiles, model-access budgets, and billing budgets as separate limits. A limit change is a reviewed configuration or infrastructure change with an owner, expiry or review date, cost approval, and post-change readback. Do not solve capacity by silently weakening containment, retention, admission, or cleanup checks.
Quick paths¶
| Need | Start here |
|---|---|
| Install, upgrade, or roll back | RB012, RB013, RB015, RB023 |
| GitHub-less GCP deploy, update, or teardown failure | RB023 |
| GCP bucket key or object re-encryption failure | RB023 |
| Back up, restore, and reconcile | RB014, RB003 |
| Queue or provider dependency outage | RB008, RB020 |
| Sensor or collector loss | RB007 |
| Quarantine or revoke credentials | RB009, RB002 |
| Stuck, cancelled, or expired range | RB001, RB002, RB017 |
| Residual inventory | RB003 |
| Capacity or cost limit | RB011, RB018, RB019 |
| Browser feature blocked by CSP | RB024 |
| Escalation ownership | Shared operating contract and the Escalate condition in each runbook |
Catalog¶
| ID | Incident |
|---|---|
| RB001 | Range provisioning failed |
| RB002 | Range teardown stuck |
| RB003 | Orphaned range resources |
| RB004 | Portal unavailable |
| RB005 | Database connection errors |
| RB006 | Worker or provider-function errors |
| RB007 | Logs or containment signals not flowing |
| RB008 | Dead-letter messages |
| RB009 | User locked out or credential revoked |
| RB010 | Agent upload failed |
| RB011 | Upload storage quota exceeded |
| RB012 | Portal installation or deployment |
| RB013 | Infrastructure deployment |
| RB014 | Database backup, restore, and reconciliation |
| RB015 | Database migrations |
| RB016 | Database maintenance window |
| RB017 | Stale range cleanup |
| RB018 | Container registry cleanup |
| RB019 | System health check |
| RB020 | Provisioner timeout |
| RB021 | High database connections |
| RB022 | AWS Firehose delivery lag |
| RB023 | GitHub-less GCP deploy, update, or teardown failed |
| RB024 | Browser feature blocked by Content Security Policy |
Detailed current-state ownership is in the architecture
handbook. Existing provider procedures under
docs/ops/ and docs/dev/ remain authoritative for their specific surfaces.