Skip to content

Operations runbooks

These runbooks are the operational handoff for the current BigRAE control plane. GCP with GCE range cells is the primary supported release path. AWS procedures are called out only where the repository and a deployed AWS tenant provide an equivalent surface. GDC VM Runtime remains an operator-validation mode and is not a live-fire containment boundary.

The procedures were checked against the repository and deployed service types on 2026-10-23. Resolve names from the selected deployment's Terraform outputs and protected environment; never copy names from an example or another tenant.

Shared operating contract

Set non-secret identifiers before using command examples:

export PROJECT_ID="<gcp-project-id>"
export REGION="<gcp-region>"
export CLUSTER="<gke-cluster>"
export NAMESPACE="<platform-namespace>"
export JOBS_NAMESPACE="<jobs-namespace>"
export REQUEST_ID="<request-uuid>"
export OPERATION_ID="<operation-uuid>"
export SINCE="2h"

For AWS-only branches, use the approved profile and set AWS_REGION and AWS_CLUSTER. Confirm the active cloud identity, selected project/account, cluster context, namespace, incident window, and operator authorization before investigation. Production is never an implicit default.

Investigation is read-only. Keep request and operation IDs in evidence, but redact user data, tenant identifiers, provider payloads, signed URLs, database connection strings, and credential material. Never print Kubernetes Secrets, Secret Manager values, process environments, or complete Terraform outputs.

Use the application service boundary for range, account, upload, and credential changes. Use the protected deployment workflow for infrastructure, image, migration, or workload changes. A GCP deployment without GitHub uses the release-driven gcp-deploy command instead (see the GitHub-less GCP deployment guide). Direct SQL, hand-built events, hand-run production Terraform or Helm, arbitrary kubectl exec, and deleting cloud resources by name are not recovery paths.

Escalate when the safe mutation surface is absent, ownership is ambiguous, cleanup cannot be proven, a security boundary may have failed, or recovery would change infrastructure outside the selected deployment. The incident commander owns service restoration; the security lead owns containment and credential revocation; the platform owner owns deployment and Terraform; the data owner approves restore/cutover; the range owner confirms lifecycle and residual inventory.

Treat application quotas, cloud service quotas, event-capacity profiles, model-access budgets, and billing budgets as separate limits. A limit change is a reviewed configuration or infrastructure change with an owner, expiry or review date, cost approval, and post-change readback. Do not solve capacity by silently weakening containment, retention, admission, or cleanup checks.

Quick paths

Need Start here
Install, upgrade, or roll back RB012, RB013, RB015, RB023
GitHub-less GCP deploy, update, or teardown failure RB023
GCP bucket key or object re-encryption failure RB023
Back up, restore, and reconcile RB014, RB003
Queue or provider dependency outage RB008, RB020
Sensor or collector loss RB007
Quarantine or revoke credentials RB009, RB002
Stuck, cancelled, or expired range RB001, RB002, RB017
Residual inventory RB003
Capacity or cost limit RB011, RB018, RB019
Browser feature blocked by CSP RB024
Escalation ownership Shared operating contract and the Escalate condition in each runbook

Catalog

ID Incident
RB001 Range provisioning failed
RB002 Range teardown stuck
RB003 Orphaned range resources
RB004 Portal unavailable
RB005 Database connection errors
RB006 Worker or provider-function errors
RB007 Logs or containment signals not flowing
RB008 Dead-letter messages
RB009 User locked out or credential revoked
RB010 Agent upload failed
RB011 Upload storage quota exceeded
RB012 Portal installation or deployment
RB013 Infrastructure deployment
RB014 Database backup, restore, and reconciliation
RB015 Database migrations
RB016 Database maintenance window
RB017 Stale range cleanup
RB018 Container registry cleanup
RB019 System health check
RB020 Provisioner timeout
RB021 High database connections
RB022 AWS Firehose delivery lag
RB023 GitHub-less GCP deploy, update, or teardown failed
RB024 Browser feature blocked by Content Security Policy

Detailed current-state ownership is in the architecture handbook. Existing provider procedures under docs/ops/ and docs/dev/ remain authoritative for their specific surfaces.