AWS Terraform apply order¶
Part of the BigRAE deploy and operations docs; start at the documentation home.
This is the authoritative order for applying the AWS Terraform stacks on a fresh account, the backend state key each stack uses, and the values you must override before the first apply. Follow it top to bottom. The CI Deploy workflow applies the same stacks in the same order; this doc lets an operator reproduce a first apply by hand and understand what CI does.
For the secrets and repository variables that back these values, see
deploy-secrets.md. For range AMIs, see
aws-ami-seeding-runbook.md. To tear an account
down, see aws-teardown-runbook.md.
Prerequisites¶
Before any stack applies, these must already exist:
- Bootstrap has run for the environment
(
scripts/bootstrap/deploy.py bootstrap --env <env> --profile <profile>), creating the S3 state bucket, the GitHub OIDC provider, the deploy IAM role, and the per-instance backend config files under~/.shifter/<env>-<bucket>/. - Self-hosted runners are provisioned and registered if you deploy through
CI (all AWS deploy jobs use
runs-on: self-hosted). Runscripts/bootstrap/deploy.py runners --env <env> --profile <profile>to provision + auto-register them (issue #1433). Seeaws-runner-provisioning-runbook.md. - Range AMIs are built and the
/shifter/ami/{kali,ubuntu,windows,dc}SSM parameters are seeded. The Portal stack reads these as data sources and its plan fails if any is missing. Seeaws-ami-seeding-runbook.md. - The engine provisioner image is built (CI
_shifter-engine.yml) so the Portal stack can resolve its engine image digest. Locally, the digest comes from the same ECR repo the engine build pushes to.
Terraform version: the stacks require >= 1.5.0 with the hashicorp/aws
provider ~> 6.0; CI pins Terraform 1.13.3.
Stacks, order, and state keys¶
Apply in this order. Each stack initializes against the shared state bucket with
its own -backend-config=<env>.s3.tfbackend (all use region = us-east-2,
encrypt = true, use_lockfile = true). Replace the placeholder bucket in the
tfbackend with the bootstrap-created state bucket, or use the rendered config
under ~/.shifter/<env>-<bucket>/.
| Order | Stack | Directory | State key |
|---|---|---|---|
| 1 | Core | platform/terraform/environments/<env>/ |
shifter/<env>/terraform.tfstate |
| 2 | Range | platform/terraform/environments/<env>/range/ |
<env>/range/terraform.tfstate |
| 3 | Engine image | (no Terraform stack) | built by _shifter-engine.yml into ECR |
| 4 | Portal | platform/terraform/environments/<env>/portal/ |
<env>/portal/terraform.tfstate |
Only the Core key carries the shifter/ prefix. The Portal stack reads the Core
state (shifter/<env>/terraform.tfstate) and the Range state
(<env>/range/terraform.tfstate) as remote-state data sources, so these keys
must match exactly.
There is no separate Engine Terraform stack.
The provisioner image is built before the control-plane release.
Provisioner workloads run as Kubernetes Jobs on EKS.
The isolated environments/<env>/eks root owns cluster infrastructure.
The shared Helm chart owns application workloads.
Follow the AWS EKS bundle after preparing the data-plane stacks.
The retained production roots have owner @Brad-Edwards and a local terraform/full bootstrap caller.
Per-stack first-apply overrides¶
The committed terraform.tfvars files are an example.com baseline that is
intentionally non-operational where a real account value is required. Supply the
real values in a gitignored local.auto.tfvars next to each stack's
terraform.tfvars (local), or via the TF_VARS_<ENV>_* secrets that CI renders
into local.auto.tfvars (see deploy-secrets.md).
Core (environments/<env>/)¶
budget_alert_email: real operations address for the S3 budget alert. Seelocal.auto.tfvars.examplein the stack directory.
Range (environments/<env>/range/)¶
agent_s3_bucket: the account-specific user-storage bucket range instance roles read (baseline placeholderREPLACE_AGENT_S3_BUCKET).vm_series_ami_id: the regional PAN-OS VM-Series Marketplace AMI (baseline placeholderREPLACE_VM_SERIES_AMI_ID).victim_allowed_cidrs: not set here. It is rendered intovictim_allowed_cidrs.auto.tfvarsfrom the deployment'sbigrae.yaml(settings.range_egress) bybigrae-config render. Keep it out of the whole-file range secret. See../architecture/range-egress-ip-allowlist.md.
Portal (environments/<env>/portal/)¶
domain_name,ses_domain,ctfd_domain,ctf_from_email: real DNS and mail identities (baseline isexample.com).alarm_email: real operations address.allowed_email_domains: sign-up allow-list. Empty in the baseline, which fails closed (no one can self-register).cognito_domain_prefix: must be globally unique across AWS Cognito.user_storage_bucket: account-suffixed bucket name (baseline placeholdershifter-<env>-user-storage-REPLACE_WITH_ACCOUNT_ID).ctfd_ssh_public_key,ctfd_ssh_allowed_cidrs: empty in the baseline, which leaves CTFd SSH ingress closed. Set both to enable it.enable_portal_inspection:falsein the baseline. Settrueto route ALB and portal traffic through the inspection firewall (the apply then asserts the routed path is healthy).
The Portal stack also depends on the /shifter/ami/* SSM parameters and the
engine image digest existing before plan; both come from the prerequisites
above.
The application deployment is digest-first through EKS.
Publish the attested images before invoking eks-deploy.
The retained Portal stack supplies persistent services; it does not start application containers on EC2.
Applying¶
Local, per stack (repeat for Core, then Range, then Portal):
cd platform/terraform/environments/<env>/<stack>
terraform init -backend-config=<env>.s3.tfbackend
terraform plan
terraform apply
scripts/bootstrap/deploy.py terraform --env <env> --profile <profile> runs the
same core -> range -> portal order and captures the Portal remote-state
tfvars. Never apply prod locally; use the Deploy workflow.
The first Portal apply blocks on DNS-validated ACM certificates and SES
identities. Publish the validation records while the apply is waiting, as
described in the first-run DNS validation section of
deploy-secrets.md.