RB019: Check system health¶
Applicability: GCP primary; AWS optional. Operator: platform on-call. Verified: 2026-10-23.
Symptom¶
An operator needs a rapid health assessment before or during an incident, deployment, event, or capacity change.
Investigation¶
Collect each independent signal; do not collapse them into one green status.
- Public: DNS, certificate, ingress/backend health, and
/health/. - Cluster: node/control-plane status through the approved private access path; rollout, pod readiness, restarts, events, and resource pressure.
- Workloads:
portal-web, launcher, result applier, outbox drainer, reconciler, CMS/Engine/Mission Control workers, Guacamole, OpenFGA, Redis, and database. - Queues: Pub/Sub or SNS/SQS backlog, oldest age, dead letters, and worker heartbeats.
- Release: exact running image digests, runtime-config checksum, latest deployment readback, migration result, and post-deploy smoke.
- Range plane: provider quota, subnet/NAT capacity, active/stuck operations, cleanup verification, residual inventory, application/model budgets, billing budget alerts, and approved event-capacity limits.
Read-only command:
kubectl -n "$NAMESPACE" get deploy,pods
kubectl -n "$JOBS_NAMESPACE" get jobs \
-l app.kubernetes.io/component=pulumi-provisioner
kubectl -n "$NAMESPACE" get events --sort-by=.lastTimestamp
Resolution¶
Open the narrow runbook for the failed signal. Restore the owning dependency or redeploy an exact verified release through the protected workflow. A healthy process, HTTP response, Terraform apply, or smoke check proves only its own boundary; keep the incident open until the failed user/operator path and independent cleanup or containment evidence recover.
Prevention¶
Maintain a dashboard with the independent signals above, synthetic public and lifecycle probes, capacity/cost headroom, and release identity. Run the functional smoke for release qualification, not as a replacement for continuous health checks.
Escalate on cross-layer disagreement, unknown running artifacts, containment signal loss, exhausted quota/budget, or provider inventory without ownership.