Disaster Recovery Runbook—AWS Portal Stack¶
Tracking issue: https://github.com/Brad-Edwards/bigrae/issues/160
Design note: docs/architecture/aws-disaster-recovery-preflight-160.md
Region: us-east-2 (all portal resources)
This runbook is the operator procedure for recovering portal data and infrastructure after a destructive failure (accidental delete, corruption, region/AZ outage, failed deploy). It covers the AWS portal stack only—RDS, the portal upload bucket, Cognito, Terraform state and engine/Pulumi state. AWS runtime recovery uses the shared chart on EKS. See EKS operations for cluster and workload recovery. GCP disaster recovery remains outside this runbook.
How to read this runbook¶
Three concepts are kept distinct on purpose—conflating them produces a DR plan that looks complete but cannot restore data:
- Availability controls (Multi-AZ, pod replacement, health checks) reduce downtime from a single failure. They are not a tested restore and do not protect against delete/corruption that propagates to the standby.
- Recoverability controls (backups, snapshots, object versions, identity exports) are what you restore from. These are the DR primitives.
- Evidence is a dated record that a restore primitive was actually exercised and met its target. An untested backup is an assumption, not a control. See Tested recovery exercise.
Data classes drive the target. Authoritative state (RDS) must be restorable with a bounded RPO. Generated state (Kubernetes workloads, Redis, range compute) is rebuilt, not restored. Ephemeral state (portal uploads) is accepted-loss by design. Identity-provider state (Cognito) is reseeded, not restored into the app database. Infrastructure state (Terraform/Pulumi) is recovered from object versions.
Component recovery matrix¶
| Component | Data class | Owner surface | RTO | RPO | Restore primitive | Alert source | Region policy |
|---|---|---|---|---|---|---|---|
| Portal RDS (PostgreSQL) | Authoritative | modules/portal/rds, env db_* vars |
≤ 2 h (drill: ~53 min) | ~5–10 min (PITR log lag) | PITR or snapshot restore to new instance | RDS event subscription (backup/failure/low storage) → backup-alerts SNS |
Same-region; Multi-AZ in prod. Cross-region copy deferred (see Deferred) |
| Guacamole RDS (PostgreSQL) | Authoritative (session/connection metadata) | modules/portal/eks, Guacamole secret and database settings |
≤ 2 h | ~5–10 min (PITR log lag) | PITR or snapshot restore to new instance | RDS event subscription → backup-alerts SNS | Same-region; Multi-AZ in prod |
| Portal uploads (S3) | Ephemeral | modules/portal/s3 |
n/a | Accepted loss | None—rebuild on re-upload | n/a | Same-region. Versioning/CRR intentionally off (see S3 uploads) |
| Platform workloads | Generated | platform/charts/shifter, EKS root |
Requalify after restore | n/a (stateless) | Restore prerequisites, then atomic Helm deployment | Kubernetes readiness and rollout checks | Same-region EKS |
| Cognito user pool | Identity-provider | modules/portal/cognito |
≤ 4 h | Best-effort (export cadence) | Terraform recreate pool + reseed users; Django users restore via RDS | Pool deletion protection (prod) | Same-region; managed service |
| Terraform state (S3 backend) | Infrastructure | scripts/bootstrap/terraform_backend.py, render_aws_backend_configs.py |
≤ 1 h | ≤ last apply | S3 object-version restore; cross-region copy of state bucket | S3 bucket versioning | Same-region; cross-region copy is an operator step |
| Engine/Pulumi state (S3 + DynamoDB) | Infrastructure | modules/engine-state |
≤ 1 h | ≤ last stack op | S3 noncurrent-version restore; DynamoDB PITR for locks | DynamoDB PITR | Same-region |
| Secrets Manager | Authoritative (credentials) | Portal and EKS roots, runtime secret references | ≤ 30 min | n/a (regenerated) | Terraform regenerates from random_password; rotate consumers |
n/a | Same-region |
| Redis (channel layer) | Generated | modules/portal/redis |
≤ 30 min | n/a (ephemeral) | Recreate replication group; clients reconnect | Existing Redis alarms | Same-region |
| Range compute/state | Generated | engine provisioner, Pulumi state | ≤ 1 h per range | n/a | Re-provision from scenario template + engine/Pulumi state | Range launch-failure alarm | Same-region |
RTO/RPO are targets, not guarantees. The portal-RDS row is backed by a real PITR drill (see Tested recovery exercise—2026-06-24: ~53 min RTO, ~7 min RPO on dev); the RPO is bounded by RDS transaction-log shipping, so confirm the instance's latest restorable time during an incident rather than assuming it. The other rows are design targets whose evidence log is still to be filled—a target is only fully credible once that component's drill has run.
Per-component recovery procedures¶
Portal RDS (PostgreSQL)¶
The portal database is the authoritative application store (Django
auth_user, UserProfile, audit trail, CMS/CTF/range metadata). Automated
backups are enabled (backup_retention_days = 7 in all environments),
copy_tags_to_snapshot = true, and a final snapshot is taken on delete except
where db_skip_final_snapshot = true (dev).
RDS restores never overwrite the source instance—they create a new instance. Recovery is therefore: restore to a new identifier, validate, then repoint the application by updating the DB host (Secrets Manager + SSM) or by renaming.
Point-in-time recovery (preferred—bounded RPO):
# 1. Confirm the window actually available right now.
aws --region us-east-2 rds describe-db-instances \
--db-instance-identifier <name_prefix>-db \
--query 'DBInstances[0].LatestRestorableTime'
# 2. Restore to a new instance at the chosen timestamp.
aws --region us-east-2 rds restore-db-instance-to-point-in-time \
--source-db-instance-identifier <name_prefix>-db \
--target-db-instance-identifier <name_prefix>-db-restore \
--restore-time 2026-06-24T12:00:00Z \
--db-subnet-group-name <name_prefix>-db-subnet \
--no-publicly-accessible
# 3. Wait until available, then validate row counts / latest audit timestamp
# against the expected pre-incident state before any cutover.
aws --region us-east-2 rds wait db-instance-available \
--db-instance-identifier <name_prefix>-db-restore
Snapshot restore (when PITR window has aged out): use
restore-db-instance-from-db-snapshot with the latest automated or the final
snapshot (<name_prefix>-db-final). Same new-instance-then-cutover flow.
Cutover: the application reads the DB host from the
shifter-<name_prefix>-db-credentials Secrets Manager secret. After validating
the restored instance, update that secret's host to the restored endpoint and
redeploy (SSM) so workers reconnect, OR delete the corrupted instance and rename
the restored one to the original identifier. Prefer the secret-update path in
prod so you keep the corrupted instance for forensics.
Guacamole RDS (PostgreSQL)¶
Same primitives and flow as portal RDS, against <name_prefix>-guacamole-db and
the shifter-<name_prefix>-guacamole-db secret. Guacamole stores connection
definitions and session/recording metadata—authoritative for Guacamole, but
distinct from the portal DB. A portal DB recovery does not recover Guacamole;
run both if both were affected.
Portal uploads (S3)¶
Classified ephemeral by design—authoritative state lives in RDS, so the
upload bucket has versioning and cross-region replication intentionally disabled
(see docs/architecture/s3-bucket-hardening-preflight.md
and #143/#219). There is no restore primitive: lost upload objects are
re-uploaded by users/agents. Do not silently flip this bucket to durable to
make a generic checklist pass—that changes delete/retention semantics for
every consumer (agent uploads, experiment scripts, CTF attachments) and requires
revisiting lifecycle, KMS, and ADR exceptions as its own design decision.
Platform workloads on EKS¶
Application state lives in managed databases and object stores.
Recreate failed cluster infrastructure through the isolated EKS root.
Restore data and secret references before deploying workloads.
Use attested image digests with the normal eks-deploy lifecycle.
Verify migrations, readiness, workers and authenticated remote access after recovery.
Do not use retired EC2 portal scripts for workload recovery.
Follow EKS operations for deployment checks and rollback limits.
Cognito¶
The user pool is Terraform-managed (config recreatable from
modules/portal/cognito), but user identities, MFA enrollment, and group
membership are not in RDS. Pool deletion protection is on in prod.
- If the pool config is lost but the pool survives:
terraform applyrestores configuration (clients, domain, triggers). - If user identities are lost: there is no point-in-time restore. Reseed from
the last user export (see cadence in Deferred) or
re-invite. Django
auth_user/UserProfile/groups are restored from RDS and are re-synchronized from verified provider claims at next login—so the RDS restore covers app-side authorization; the Cognito reseed covers authentication identity. Treat these as two separate restores.
Terraform state (S3 backend)¶
State lives in an S3 backend (encrypt = true, use_lockfile = true,
us-east-2); bucket/key are injected at init via -backend-config, rendered
by scripts/bootstrap/terraform_backend.py /
scripts/terraform/render_aws_backend_configs.py. The state bucket has
versioning enabled.
- Corrupted/deleted state object: restore the prior S3 object version of the
specific
…/terraform.tfstatekey (aws s3api list-object-versions→copy-objectthe good version back over the key). Theuse_lockfilesetting means no separate DynamoDB lock to reconcile. - Region/bucket loss: copy the versioned state bucket to a recovery bucket
(
aws s3 syncor cross-regioncopy-object) and re-initwith a-backend-configpointing at the recovery bucket. Do not commit live backend bucket/key names into the repo—keep rendered backend configs in~/.shifteror CI temp space as the bootstrap script already does.
State may contain generated secret material; treat recovered state files with the same handling as secrets (no world-readable copies, no CI artifacts).
Engine / Pulumi state (S3 + DynamoDB)¶
The engine stores range/stack state in the <name_prefix>-pulumi-state-<account>
S3 bucket (versioned, force_destroy = false) with a DynamoDB locks table
(<name_prefix>-pulumi-locks, PITR enabled) and the <name_prefix>-pulumi-secrets
CMK.
- State object loss: restore the prior S3 object version (noncurrent versions transition to Glacier after the configured window—restore from Glacier first if the needed version has aged out).
- Lock table loss/corruption: restore via DynamoDB point-in-time recovery to a new table, then repoint the engine, or recreate the table (locks are transient).
- Range recovery: ranges are generated compute. Re-provision from the scenario template; the engine reconciles against recovered Pulumi state.
Secrets Manager¶
DB and app credentials are generated by Terraform (random_password) and stored
in Secrets Manager under the portal CMK. The portal DB credentials secret uses
recovery_window_in_days = 0 (immediate deletion on destroy, to avoid
name-collision on recreate); Guacamole uses guacamole_secrets_recovery_window_days
(7 in prod, 0 in dev). Implication: there is no recovery window to restore a
deleted secret—instead terraform apply regenerates the secret, after which
consumers must pick up the new value (SSM redeploy / restart). Never print secret
values into logs, argv, state exports, or CI artifacts during recovery.
Redis (channel layer)¶
The Django Channels Redis layer is generated/ephemeral—no durable state. Recreate the replication group via Terraform; clients reconnect automatically. In-flight WebSocket/terminal sessions are lost and re-established by the browser.
Tested recovery exercise¶
DR targets are only credible once a real restore has been run end-to-end. The canonical exercise is an RDS point-in-time restore-to-new-instance drill, which exercises the highest-value authoritative store without touching the live instance (PITR always restores to a new instance).
Procedure (run against a non-production environment, for example dev):
- Note a known-good marker in the source DB (for example latest
auditrow id + timestamp) and the current timeT. - Run the PITR command from Portal RDS targeting
<name_prefix>-db-drill, restore-timeT. - Record wall-clock from command start to
db-instance-available→ observed RTO. - Connect to the restored instance, confirm the marker row is present and no
newer-than-
Trows exist → observed RPO (gap betweenTand latest restorable time). - Delete the drill instance (
delete-db-instance --skip-final-snapshot) so the exercise leaves no billable resource. - Fill the evidence log below.
Evidence log:
| Date | Operator | Component | Restore primitive | Observed RTO | Observed RPO | Result | Artifact |
|---|---|---|---|---|---|---|---|
| 2026-06-24 | Platform operator (dev account) | Portal RDS (dev) dev-portal-db |
PITR (--use-latest-restorable-time) → new dev-portal-db-drill (db.t3.medium, single-AZ) |
~53 min to available (endpoint live ~3 min in; remainder is the post-restore initial automated backup) |
~7 min (source latest-restorable 02:15:17Z vs restore decision 02:22:07Z) |
PASS—restored instance reached available, engine 16.13 / 100 GB / encrypted matching source, with its own restorable window. Validated at infrastructure level; in-DB row-count validation not performed (private endpoint, no in-VPC session wired for the drill). Drill instance deleted (--skip-final-snapshot --delete-automated-backups). |
AWS RDS event history for dev-portal-db-drill; issue #160 thread |
Notes from the 2026-06-24 run: observed RTO (~53 min) is comfortably within the ≤ 2 h target, but most of it is the initial automated backup RDS takes after the restore—the instance is connectable far earlier (~3 min). In a real incident you can begin validation/cutover before the clean
availablesignal. Observed RPO (~7 min) reflects normal RDS transaction-log shipping lag (latest restorable time trails ~5–7 min behind now); treat the ≤ 5 min PITR target as a floor and expect a few minutes of lag in practice.The drill requires AWS access and creates a temporary billable instance. Re-run it periodically (and after major schema/engine changes) and append a new row.
Backup-failure monitoring¶
Backup and snapshot failures surface as RDS events, not CloudWatch metrics,
so detection uses an RDS event subscription rather than a synthetic metric.
The modules/portal/backup-alerts module (wired in every portal env root)
creates:
- a dedicated customer-managed KMS key whose policy grants the
events.rds.amazonaws.comservice principalkms:GenerateDataKey*+kms:Decrypt(account-scoped). This is required because the sharedaws_sns_topic.alertstopic is encrypted with the AWS-managedalias/aws/snskey, whose policy cannot grant that principal—RDS delivery to it would silently fail. - a CMK-encrypted SNS topic
<name_prefix>-db-backup-alertswith a topic policy allowing the RDS event service to publish, and an email subscription gated onalarm_email. - one RDS event subscription covering the portal and Guacamole DB instances,
categories
availability,backup,failure,low storage,maintenance,recovery.
To receive alerts: set a real alarm_email in the environment's
terraform.tfvars and confirm the SNS email subscription (AWS sends a
confirmation link on first apply). To verify delivery without waiting for a
real failure, publish a test message to the topic
(aws sns publish --topic-arn <topic-arn> --message "DR alert test") and confirm
the email arrives.
Deferred enhancements¶
These are evaluated and intentionally not implemented now, recorded here so the decision is explicit rather than an unexamined gap.
- Cross-region RDS automated-backup replication. Not enabled. When an
environment needs cross-region RPO, add
aws_db_instance_automated_backups_replicationin the destination region (requires a second provider alias for that region, a destination KMS key, andsource_db_instance_arnset to the portal instance), and add areplicate_backups_to_regionvariable to gate it per environment. Trigger to enable: a stated cross-region RTO/RPO requirement for prod. - AMI backups for EC2. Rejected—instances are stateless/generated and rebuilt from Terraform + ECR + SSM. Revisit only if a concrete irreplaceable host asset appears.
- S3 uploads versioning / cross-region replication. Deferred per #143/#219; the bucket is ephemeral by design. Enabling CRR requires versioning, which changes delete/retention semantics—a separate design decision, not a DR toggle.
- Cognito scheduled user export. No automated export cadence today; user identity reseed is best-effort. Add a scheduled export job if a bounded identity RPO becomes a requirement.
References¶
- Design note:
docs/architecture/aws-disaster-recovery-preflight-160.md - S3 hardening decision:
docs/architecture/s3-bucket-hardening-preflight.md - AWS—encrypted SNS for service events: https://docs.aws.amazon.com/sns/latest/dg/sns-key-management.html
- AWS—RDS event subscriptions: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_Events.Subscribing.html