AWS range AMI seeding¶
Part of the BigRAE deploy and operations docs; start at the documentation home.
The Portal Terraform stack reads the range base AMIs from SSM as data sources
(/shifter/ami/kali, /shifter/ami/ubuntu, /shifter/ami/windows,
/shifter/ami/dc). If any parameter is missing, terraform plan for the Portal
stack fails. On a fresh account you must build these four AMIs and seed the SSM
parameters before the first Portal apply. The runtime provisioner also resolves
guest AMIs from the same /shifter/ami/* parameters.
Packer sources live in shifter/packer/ (kali.pkr.hcl, ubuntu.pkr.hcl,
windows.pkr.hcl, dc.pkr.hcl, shared variables.pkr.hcl, per-env
dev.pkrvars.hcl). The deep reference is
shifter/packer/README.md.
Ordering: no circular dependency with the portal VPC¶
Packer needs a VPC and subnet to launch its builder instance, but it does not
need the portal VPC. vpc_id and subnet_id in variables.pkr.hcl have no
defaults; the committed dev.pkrvars.hcl ships only placeholders
(vpc-xxxx / subnet-xxxx) so no live account id is tracked in the public repo.
The real builder network is supplied at build time: the CI path reads the
PACKER_BUILD_{VPC,SUBNET}_ID_<ENV> repository variables (below), and a local
build takes them from an uncommitted operator override. Either way the build uses
an existing general-purpose VPC/subnet, independent of the portal Terraform that
has not been applied yet. The Portal stack consumes the AMIs only as SSM data
sources, which is a soft, one-directional dependency.
The correct order on a fresh account is therefore:
- Bootstrap the account.
- Provision runners (for the CI build path).
- Build the AMIs and seed
/shifter/ami/*(this runbook). - Apply the Portal stack.
There is no chicken-and-egg between packer and the portal VPC. Do not try to point packer at the portal VPC; use the account default VPC or an operator-owned public subnet with outbound internet egress.
Prerequisites¶
- Kali Marketplace opt-in. The Kali build sources an AWS Marketplace AMI
(product code
7lgvy7mt78lgoi4lant0znp5h). Accept the free subscription once per account at https://aws.amazon.com/marketplace/pp/prodview-fznsw3f7mq7to, or the Kali build fails withOptInRequired. - A builder VPC/subnet. An existing VPC and a public subnet with internet
egress (the account default VPC works). For the CI path set the
PACKER_BUILD_{VPC,SUBNET}_ID_<ENV>repository variables (see build path A); for a local build setvpc_id/subnet_idin an uncommitted operator override. Never commit live IDs into the tracked pkrvars. - Runners if you build through CI (the Packer workflow is
self-hosted). - The base-image build IAM role and its secret for the CI build path, one time per account (see the next section).
Base-image build IAM role (one-time cutover)¶
Since issue #1656 the base build job in packer.yml assumes a dedicated
least-privilege image-pipeline role (github_actions_image in
platform/terraform/global/iam) instead of the broad deploy role. That role
trusts only the dev/main protected-branch OIDC subjects and can pass only the
exact shifter-<env>-range-range-instance role to EC2, so a base-image
verification instance can receive only the range role. The check_tf_iam_role_naming
guardrail (ADR-004-R22) pins those invariants.
The implementation PR ships the Terraform, workflow wiring, and guardrail but does
not apply IAM or set secrets. Complete the cutover once per AWS account (dev,
and prod for promotions):
- Apply the global-IAM stack so the role and its output exist:
- Set the
AWS_IMAGE_ROLE_ARN_<ENV>GitHub secret from the output (prod is the unsuffixedAWS_IMAGE_ROLE_ARN; base builds targetdev):
ROLE_ARN=$(terraform -chdir=platform/terraform/global/iam output -raw github_actions_image_role_arn)
REPO="$(gh repo set-default --view)"
gh secret set AWS_IMAGE_ROLE_ARN_DEV --repo "$REPO" --body "$ROLE_ARN"
- Confirm the repository's OIDC subject format and immutable IDs match the Terraform inputs. The renamed canonical repository uses immutable subjects:
REPO="$(gh repo set-default --view)"
gh api "repos/$REPO/actions/oidc/customization/sub"
# expect use_default: true, use_immutable_subject: true,
# sub_claim_prefix: repo:<owner>@<owner_id>/<repo>@<repository_id>
AWS trust defaults to github_subject_format = "immutable". Other repositories
must supply their own owner and repository names and immutable IDs. Select
"default" explicitly only when the repository still emits the legacy subject
shape. Keep the existing exact dev and main image-build subjects.
Until the secret is set the base build job fails closed with
Required secret AWS_IMAGE_ROLE_ARN_<ENV> is not set. The bake-scenario job
keeps its own deploy-role secret and is unaffected.
Build path A: the Packer workflow (recommended)¶
The Packer AMI Build workflow (.github/workflows/packer.yml) is
workflow_dispatch on a self-hosted runner. It builds one AMI type per run,
extracts the AMI id from the build manifest, and writes
/shifter/ami/<type> with aws ssm put-parameter --overwrite. The build runs
only from a protected ref (dev or main); a dispatch from a feature branch is
rejected before AWS authentication, because the job executes checked-out code and
publishes SSM pointers. Since issue #1656 the base build job assumes a
dedicated least-privilege image-pipeline role, not the broad deploy role (see
Base-image build IAM role).
The base build job takes its builder VPC and subnet from trusted repository
Actions variables too, not the committed pkrvars (issue #1777). Set these once per
account (suffixed by environment, _DEV), pointing at a public subnet
with an internet-gateway route (the account default VPC is fine):
| Variable | Value |
|---|---|
PACKER_BUILD_VPC_ID_<ENV> |
VPC the Packer builder launches in (the public subnet's VPC; the account default VPC works) |
PACKER_BUILD_SUBNET_ID_<ENV> |
A public subnet in that VPC with an internet-gateway route, so SSH/WinRM reach the builder over its public IP |
They are validated and passed as -var after -var-file, so they override the
committed placeholder without any live id entering the tree. A missing variable
fails the build closed.
For kali, ubuntu, and windows the workflow first runs a fresh-boot
validation gate (issue #1633) before it seeds SSM. The gate boots the exact
built AMI in a runtime-equivalent range subnet with the range instance profile,
waits for the guest to register with SSM (registration requires the guest to
resolve the regional SSM endpoint, so a successful registration is the durable
proof that guest DNS works), resolves the endpoint through the guest system
resolver, reboots, and confirms both again. The SSM parameter is overwritten
only after the gate passes; a failed candidate leaves the previous known-good id
in place.
The gate reads its subnet, security group, and instance profile from trusted
repository Actions variables, not dispatch inputs, so the candidate's launch
identity cannot be chosen at dispatch time. Set these once per account (Settings
-> Secrets and variables -> Actions -> Variables), suffixed by environment
(_DEV):
| Variable | Value |
|---|---|
PACKER_VERIFY_SUBNET_ID_<ENV> |
A range-equivalent subnet that can reach SSM (private SSM endpoints or a NAT path), no inbound |
PACKER_VERIFY_SG_ID_<ENV> |
A no-inbound, egress-all security group |
PACKER_VERIFY_INSTANCE_PROFILE_<ENV> |
The SSM-enabled range instance profile (name ends in -range-instance), from terraform -chdir=platform/terraform/environments/<env>/range output -raw range_instance_profile_name |
Dispatch the three built base types (the gate reads the variables above):
for t in kali ubuntu windows; do
gh workflow run "Packer AMI Build" -f ami_type="$t" -f environment=dev -f ref=dev
done
dc is not built by this workflow. /shifter/ami/dc is a pre-promoted
internal.shifter Domain Controller published from the checked-in
dc-amis.json; dispatching ami_type=dc reads the environment's id from that
file and seeds SSM without a build (see "Pre-promoted DC AMI" below):
environment accepts dev. There is no prod build option: prod
AMIs are produced by promoting a validated dev AMI with packer-promote.yml, not
by building directly in prod. Private pack images are built and published from their owning repositories.
The base types above are the ones the portal plan requires.
Build path B: local Packer¶
From a workstation with Packer and AWS credentials for the target account:
cd shifter/packer
packer init .
packer validate -var-file=dev.pkrvars.hcl -only='*.kali' .
packer build -var-file=dev.pkrvars.hcl -only='*.kali' .
# Read the AMI id from the manifest and seed SSM:
AMI_ID=$(jq -r '.builds[-1].artifact_id | split(":")[1]' kali-manifest.json)
aws ssm put-parameter --name /shifter/ami/kali --type String \
--value "$AMI_ID" --overwrite --region us-east-2
Repeat for ubuntu and windows. Do not build dc locally: dc.pkr.hcl is a
generalized Windows image with the AD DS feature installed but not promoted, and
publishing it to /shifter/ami/dc would corrupt the pre-promoted DC contract.
Seed /shifter/ami/dc from dc-amis.json instead:
AMI_ID=$(jq -r '.dev' shifter/packer/dc-amis.json)
aws ssm put-parameter --name /shifter/ami/dc --type String \
--value "$AMI_ID" --overwrite --region us-east-2
Pre-promoted DC AMI¶
/shifter/ami/dc points at a manually created, pre-promoted internal.shifter
Domain Controller (domain internal.shifter, NetBIOS INTSHIFTER). The AMI ids
live in shifter/packer/dc-amis.json; no Packer source rebuilds them. Both the
build workflow (ami_type=dc) and packer-promote.yml publish the checked-in id
rather than a fresh build. Each reads dc-amis.json from a dedicated checkout of
the protected dev ref and resolves it through the shared validator
shifter/packer/scripts/bake/resolve-dc-ami.sh, which fails closed unless the id
exists, is AMI-shaped, and names an image EC2 reports as available and owned by
the target account; the prod promote job additionally runs only from a protected
ref (dev/main). See issue #1656.
When you re-bake the pre-promoted DC (see AMI management), set the DC's DNS forwarders to the link-local AmazonProvidedDNS so external names such as the regional SSM endpoint resolve deterministically:
This is the DC-role equivalent of the FallbackDNS baked into the Linux guests.
Do not apply the Linux or Windows-victim first-boot DNS reset to the DC: a
promoted DC owns its own DNS (its client points at itself and the DNS Server role
forwards outbound queries), so resetting the adapter to DHCP DNS would break
domain resolution.
Verify¶
Confirm all four parameters exist before the Portal apply:
for t in kali ubuntu windows dc; do
aws ssm get-parameter --name "/shifter/ami/$t" --region us-east-2 \
--query 'Parameter.Value' --output text
done
Every command must print an ami-... id. A missing parameter fails the Portal
terraform plan.