AWS tenant standup gotchas¶
Part of the BigRAE deploy and operations docs; start at the documentation home.
A running log of operator gotchas hit while standing up or re-pointing an AWS
BigRAE tenant that are not fixed in code, only in operator procedure. Fixes
that belong in the scripts, workflows, or Terraform go into the code, not here.
Use this alongside the authoritative runbooks:
deploy-secrets.md,
aws-terraform-apply-order.md,
aws-runner-provisioning-runbook.md,
aws-ami-seeding-runbook.md, and
scripts/bootstrap/README.md.
Re-pointing an existing tenant at a new account ("moved account")¶
When an environment (for example aws-dev) is moved to a different AWS account,
its GitHub Actions repository-level _<ENV> secrets and variables still point
at the old account. Nothing repoints them automatically; only two of them are
bootstrap-owned. Repoint the rest by hand before the first deploy, or the deploy
runs against a mix of the new role and old-account resource IDs.
Bootstrap-owned (rewritten by deploy.py bootstrap --env <env>):
AWS_ROLE_ARN_<ENV>TF_INFRA_STATE_BUCKET_<ENV>
Operator-owned, account-specific; must be regenerated/repointed for the new account:
TF_VARS_<ENV>_CORE,TF_VARS_<ENV>_PORTAL,TF_VARS_<ENV>_RANGE: these carry account-suffixed bucket names, domains, alarm email,vm_series_ami_id, etc. Regenerate from locallocal.auto.tfvarsoverlays withscripts/sync-deploy-secrets.sh --env <env>.SHIFTER_CONFIG_<ENV>_RANGE: the deploymentbigrae.yaml.AWS_IMAGE_ROLE_ARN_<ENV>: from the new account'splatform/terraform/global/iamoutputgithub_actions_image_role_arn.- Repository variables
PACKER_BUILD_{VPC,SUBNET}_ID_<ENV>andPACKER_VERIFY_{SUBNET,SG,INSTANCE_PROFILE}_<ENV>: VPC/subnet/SG IDs are account-specific. The verify instance profile name is deterministic (shifter-<env>-range-range-instance) but must exist in the new account after the range apply.
Tell the moved account apart from a fresh one by comparing a known variable
(for example PACKER_BUILD_VPC_ID_<ENV>) against the target account's actual VPC
IDs; a mismatch means the value still points at the old account.
Non-TTY / headless bootstrap needs --yes¶
deploy.py bootstrap|terraform|full run without a terminal (an automation
wrapper, CI, an agent) must pass --yes. Without it the routine proceed prompts
fall to their non-interactive fallback and the GitHub-secret and backend-config
steps are skipped. --yes never authorizes the account-recovery sweep, which
always needs its own --sweep.
Standalone runners needs TF_INFRA_STATE_BUCKET exported¶
deploy.py runners --env <env> --profile <p> (the standalone path, not full)
does not thread the bootstrap state bucket, so it resolves it from the
TF_INFRA_STATE_BUCKET environment variable. Export it to the bootstrap bucket
before running, or the runner apply fails closed with Set TF_INFRA_STATE_BUCKET
or pass the bootstrap bucket name. (The full path threads it automatically.)
First-deploy AMI ordering (Core → Range → seed AMIs → Portal)¶
A single deploy.yml dispatch runs the full chain, but Portal reads
/shifter/ami/* as data sources and the packer fresh-boot verify gate
(PACKER_VERIFY_*) needs the range instance profile + a range-VPC subnet/SG,
which only exist after the Range stack applies. So a fresh account cannot
complete Portal on the first dispatch. The order is: dispatch once (Core + Range
apply; Portal fails on missing AMIs), set PACKER_VERIFY_* from the applied
range, seed /shifter/ami/* via packer.yml, then dispatch again to complete
Portal.
PACKER_VERIFY_* values and the verify security group¶
After the Range stack applies, set the fresh-boot verify-gate variables from it:
PACKER_VERIFY_INSTANCE_PROFILE_<ENV>= range outputrange_instance_profile_name(shifter-<env>-range-range-instance).PACKER_VERIFY_SUBNET_ID_<ENV>= the range VPC'sssm-endpointssubnet (or any range-VPC subnet; the ssm/ssmmessages/ec2messages interface endpoints have private DNS and are reached VPC-locally, and the endpoint SG allows 443 from the whole VPC CIDR).PACKER_VERIFY_SG_<ENV>= a no-inbound, egress-all security group. The range stack does not provide one and locks the VPC default SG down to no egress, so create a dedicated SG in the range VPC (tag itProject=shifter,Purpose=packer-verify). This is operational AMI-pipeline infra, not range Terraform, so it does not appear in the range plan.
No AWS pre-promoted DC bake (/shifter/ami/dc)¶
/shifter/ami/dc must be a pre-promoted internal.shifter DC:
_should_promote_dc_at_runtime() is hard-disabled, so runtime promotion never
happens. AWS dc.pkr.hcl builds only a generalized Windows image (AD DS
installed, not promoted) and runs sysprep (which is incompatible with a
promoted DC), and the runbook forbids publishing it as /shifter/ami/dc. The
pre-promoted bake (a2_setup.ps1) exists only on the GCP side; the AWS DC AMIs
in shifter/packer/dc-amis.json were created manually and are account-scoped, so
they are not usable in a moved/new account. A new account therefore has no valid
DC AMI. Real fix: implement a pre-promoted DC bake for AWS (port GCP's
a2_setup promotion, drop sysprep) or manually promote+capture one, then update
dc-amis.json / seed /shifter/ami/dc. To unblock a base-range standup (which
never launches a DC) meanwhile, seed /shifter/ami/dc with a placeholder AMI so
the Portal plan's data source resolves; repoint it once a real DC is baked.
Debugging apply time Terraform bugs¶
The deploy workflow re-runs the ~25-min Quality gate on every dispatch, so
iterating apply time Terraform errors through CI is slow. For the debug/converge
phase, run the identical terraform apply locally against the same S3 backend
and rendered overlays (what _range.yml / deploy.py terraform run), fix each
error, then validate the full chain with one CI deploy.yml dispatch.
The active AWS portal runtime is EKS, not ECS/EC2¶
The legacy ECS/EC2 portal container rollout is retired: in _shifter-platform.yml
the deploy job is gated environment == '__legacy-disabled__' (never runs) and
post-deploy-smoke needs: eks-deploy. So on a fresh account the ECS/EC2 portal
Terraform applies (RDS/Cognito/SQS/SNS/S3/ALB) but the portal ALB just returns HTTP 502 (no
container rollout, no DB migration) and the range smoke never runs unless EKS is
enabled. To stand up a working tenant, enable EKS:
- Repository variable
AWS_EKS_<ENV>_ENABLED = true(drivesdeploy.yml'seks_enabled, which gateseks-deployand thuspost-deploy-smoke). - Secret
TF_VARS_<ENV>_EKS= the full EKS root var-file JSON (see below).
The ECS/EC2 portal stack is still applied and is not dead: EKS reuses its
Cognito/SQS/SNS/S3/RDS. EKS runs the containers (portal/guacamole/provisioner as
pods) pointed at those resources via runtime_env.
Building TF_VARS_<ENV>_EKS¶
The eks-deploy job feeds this JSON to the environments/<env>/eks root verbatim
(only injecting deployment_role_arn), so it must carry every required input.
There is no example file; construct it from the applied ECS/EC2 portal stack and
aws eks:
runtime_env(map): the 12 keys inshifter/installation/runtime_inventory_aws.py::AWS_EKS_REQUIRED_RUNTIME_ENV_KEYS. The provisioner topology keys (RANGE_*,*_AMI_ID,NGFW_*, ...) are NOT supplied here; theeks-provisioner-envmodule derives them from the range SSM export. Sources for the 12:AWS_REGION= the region.OIDC_AUTH_DOMAIN/OIDC_ISSUER_URL/OIDC_RP_CLIENT_ID= portal outputscognito_domain/cognito_issuer_url/cognito_client_id.OIDC_SECRET_ID= SSM/shifter/<env>/portal/cognito-secret-arn.QUEUE_{CMS,ENGINE,MC}_{CONSUMER,PUBLISHER}_ID= SSM/shifter/<env>/portal/sqs-{cms,engine,mc}-url(consumer and publisher share the one queue URL).RANGE_EVENTS_TOPIC_ID= SSM/shifter/<env>/portal/range-events-topic-id(SNS ARN).STORAGE_BUCKET_NAME= SSM/shifter/<env>/portal/s3-bucket.addon_versions(object: vpc_cni, ebs_csi, efs_csi, coredns, kube_proxy, secrets_store_csi), fromaws eks describe-addon-versions --kubernetes-version <ver>. The secrets-store add-on name isaws-secrets-store-csi-driver-provider(notaws-secrets-store-csi-driver).domain_name= the public hostname.edge_client_cidrs/provider_api_cidrs: each requires >=1 CIDR.edge_client_cidrsis the public HTTPS-ingress source allowlist (the portal is Cognito-gated).provider_api_cidrsis the pod-egress NetworkPolicy allowlist to AWS APIs; the correct value is the AWS region's public API ranges, but that is a large, drift-prone list, so a dev tenant may start with a broad CIDR and tighten later (a real hardening follow-up, not a permanent value).
Validate the whole thing before a CI cycle with a local
terraform plan -var-file=<eks.tfvars.json> (add deployment_role_arn) against
the environments/<env>/eks root using the rendered EKS backend config.
Apply platform/terraform/global/iam before enabling EKS so the deploy role can
manage the scoped EKS/OIDC resources and pass the shifter-<env>-eks-* roles.