AWS self-hosted runner provisioning¶
Part of the BigRAE deploy and operations docs; start at the documentation home.
Every AWS deploy job (_core.yml, _range.yml, _shifter-engine.yml,
_shifter-platform.yml, packer.yml) runs on runs-on: self-hosted. A fresh
AWS account has no runners, so the AWS Deploy workflow cannot run until you
provision and register at least one. Bootstrap creates the shared state backend
the runner Terraform root reuses.
This runbook is the standup entry point. The authoritative reference for runner
architecture, health monitoring, removal, and gotchas is
platform/terraform/global/github-runner/README.md.
Prerequisite¶
Run scripts/bootstrap/deploy.py bootstrap --env <env> first. It creates the S3
state bucket and rewrites platform/terraform/global/github-runner/<env>.s3.tfbackend
so the runner root uses the same backend.
Automated path (recommended)¶
scripts/bootstrap/deploy.py runners provisions and registers the fleet
end-to-end (issue #1433):
It provisions a dedicated, ADR-004-R20-compliant runner VPC by default
(create_runner_network), applies the runner root, mints a single-use token per
runner, registers each over SSM (token delivered inside one JSON --parameters
body so the operator-log redactor masks it; never written to Terraform state,
user data, a secret store, or logs), and verifies each runner online via the
GitHub API. Flags:
--use-existing-network: place the runners in an existing compliant network supplied asvpc_id/subnet_idinstead of the managed runner VPC (see step 1 below).--runner-count N: overriderunner_countfor this apply.--dry-run: show the plan without minting a token or sending SSM commands.
Once every runner shows status: online, the AWS Deploy workflow can run. The
manual steps below remain available for one-off or debugging use.
1. Choose the runner network¶
The standard placement (ADR-004-R20, issue #1437) is the dedicated runner VPC
from modules/github-runner-network: a private runner subnet with NAT-only
egress, no private-DNS interface endpoints, and encrypted flow logs. Both
dev.tfvars and proof.tfvars set create_runner_network = true, and the
runners path passes it, so a fresh account needs no network input and no live
IDs are committed (ADR-004-R14). Skip the rest of this section unless you need
an existing network.
The runner stack fails closed on the account default VPC: a range's private-DNS interface endpoints can hijack the runner's AWS API resolution. The supported alternative to the managed network is an existing compliant network, such as the portal VPC private tier. For the portal VPC, read the IDs from its outputs:
cd platform/terraform/environments/<env>/portal
terraform output vpc_id
terraform output private_subnet_ids
Put vpc_id and subnet_id in a gitignored
platform/terraform/global/github-runner/local.auto.tfvars, never in the
committed tfvars, and run deploy.py runners with --use-existing-network. That
flag passes -var=create_runner_network=false after -var-file=<env>.tfvars.
Setting create_runner_network = false in local.auto.tfvars alone has no
effect, because -var-file values override *.auto.tfvars. The subnet needs
outbound egress for GitHub, ECR, SSM, and AWS APIs.
allow_default_vpc is a narrow exception, not a placement option. It requires a
documented risk acceptance in docs/adr/exceptions.yaml with owner, scope,
reason, and expiry. No tracked environment sets it, and it's never a recovery
fallback for a failed managed-network apply. See the placement preflight:
docs/architecture/github-runner-placement-preflight-1437.md.
2. Terraform inputs¶
Defined in platform/terraform/global/github-runner/variables.tf:
| Variable | Required | Default | Notes |
|---|---|---|---|
create_runner_network |
no | false |
Standard placement (#1433, #1437). true provisions a dedicated non-default runner VPC (NAT-only egress, no private-DNS endpoints) and places the runner in it; takes precedence over vpc_id/subnet_id/allow_default_vpc. dev.tfvars / proof.tfvars and the runners path set true. |
runner_network_cidr |
no | 10.20.0.0/24 |
CIDR for the dedicated runner VPC when create_runner_network = true. |
allow_default_vpc |
no | false |
ADR-004-R20 exception requiring a documented risk acceptance. true accepts default-VPC placement and auto-resolves the default VPC + a subnet. No tracked environment sets it; no effect when create_runner_network = true. |
vpc_id |
conditional | "" |
Existing compliant network (for example the portal private tier) when create_runner_network = false. Gitignored override only. |
subnet_id |
conditional | "" |
As above; a private subnet of vpc_id with GitHub/ECR/SSM/AWS egress. |
runner_count |
no | 2 |
dev.tfvars / proof.tfvars set 3. |
instance_type |
no | t3.large |
Amazon Linux 2023, SSM access, no inbound. |
region |
no | us-east-2 |
|
alarm_email |
no | empty | Subscribe an inbox to the runner-alerts SNS topic. |
enable_system_auto_recovery |
no | true |
EC2 auto-recover on system status-check failure. |
cpu_alarm_threshold |
no | 95 |
Sustained-CPU hang-proxy alarm. |
Sizing: start with runner_count = 3 at t3.large. That covers the parallel
Core / Range / Engine / Platform jobs of one deploy with headroom; raise
runner_count if deploys queue on runner availability.
3. Apply the runner root¶
export AWS_PAGER=""
./scripts/runner-deploy.sh # init + plan
./scripts/runner-deploy.sh --apply # init + apply
The script reads PANW_SHIFTER_DEV_PROFILE. A fresh apply leaves each host
running the runner service but unregistered (RunnerServiceActive = 0) until
step 4.
4. Register each runner¶
Registration needs a single-use registration token from GitHub, exchanged once for long-lived credentials on the instance. Mint one token per runner:
export AWS_PROFILE=aws-dev
export AWS_PAGER=""
INSTANCE=i-xxxxxxxxxxxxxxxxx
NAME=shifter-github-runner-N
REPO="$(gh repo set-default --view)"
TOKEN=$(gh api -X POST "/repos/$REPO/actions/runners/registration-token" --jq .token)
aws ssm send-command \
--instance-ids "$INSTANCE" \
--document-name AWS-RunShellScript \
--region us-east-2 \
--parameters "commands=[
\"set -ex\",
\"cd /home/ec2-user/actions-runner\",
\"sudo -u ec2-user ./config.sh --url https://github.com/$REPO --token $TOKEN --labels self-hosted,linux,X64 --unattended --replace --name $NAME\",
\"./svc.sh install ec2-user\",
\"./svc.sh start\"
]"
Registration tokens are single-use and expire in about an hour; mint a fresh one per runner. Verify the fleet is online:
REPO="$(gh repo set-default --view)"
gh api "repos/$REPO/actions/runners" --jq '.runners[] | {name, status}'
Once every runner shows status: online, the AWS Deploy workflow can run. To
remove or replace a runner, see the "Removing a runner" section of the
github-runner README.
Move an existing fleet onto the managed network¶
A runner applied on the account default VPC or another network moves when its
root is next applied with create_runner_network = true. The instance subnet
changes, so Terraform replaces every runner instance, and each environment
gains one NAT gateway, an Elastic IP, and flow-log storage. Treat the apply as a
migration:
- Drain active self-hosted jobs, and don't dispatch deploys during the move. Don't run the apply from a job on the fleet it replaces.
- Keep the current state and tfvars for rollback. Check NAT gateway and Elastic IP quotas in the target region. Re-registration mints real registration tokens. The current SSM transport carries the token in AWS CLI argv and retained Run Command input; see the secret-transport gap in the placement preflight before running the move.
- Apply and re-register from outside the fleet:
./scripts/bootstrap/deploy.py runners --env <env> --profile <profile>. Replacement runners register with--replace, which takes over stale registrations of the same name. - Verify each runner's effective placement and readiness separately. Check
that each instance's subnet is the managed runner subnet, that GitHub shows
the runner
online, and that theShifter/RunnerHealthalarms clear. Then run an ordinary deploy. A running service alone doesn't prove DNS, egress, or job readiness.
The managed network is single-AZ. A NAT gateway or AZ failure stops the fleet until a re-apply restores it; that's recovery work, not high availability.