CI/CD Pipeline¶
How GitHub Actions validates and deploys BigRAE.
Overview¶
All CI/CD runs through GitHub Actions. The main orchestrator is deploy.yml,
which coordinates:
- Quality - Linting, tests, security scanning
- Core - ECR repositories (foundation)
- Range - Range VPC infrastructure
- BigRAE Engine - Container build
- BigRAE Platform - Application infrastructure, containers, deployment
Trigger Rules¶
Environment deploys are manual: run the Deploy workflow and pick the
environment input. Push and pull_request run validation only; no branch name
triggers a deploy (#730).
| Event | What Runs |
|---|---|
| Pull request | Quality only; no deploy |
Push to dev / main |
Quality only; no deploy |
Manual dispatch, environment: aws-dev |
Quality, then AWS dev deploy |
Release Please release on main |
GCP release (release-gcp.yml) for the tagged commit; no deploy |
Manual dispatch of packer-gcp.yml on dev or main |
Build, validate, and publish one GCP guest image; no deploy |
No GCP tenant deploys through GitHub Actions after #2668. GCP tenants install
published releases with the GitHub-less commands (ADR-070). The GCP deploy jobs
stay in deploy.yml, but no environment option selects them; #2672 decides
the future of that path.
Run a deploy from the Actions UI (Deploy → Run workflow, pick the branch to
deploy and the environment) or the CLI:
The branch you run it from is the code that deploys; the environment input
picks where it goes. dev is the integration branch (Quality only).
Workflow Files¶
.github/workflows/
├── deploy.yml # Main orchestrator
├── _quality.yml # Linting, tests, Checkov
├── _core.yml # ECR repositories
├── _range.yml # Range VPC
├── _gcp-dev.yml # GCP CI wrapper over the gcp-deploy commands
├── _shifter-engine.yml # Shifter Engine container
├── _shifter-platform.yml # Shifter Platform infra + deploy
├── packer-gcp.yml # GCP guest image build, validation call, and publication
├── packer-gcp-validate.yml # GCP guest candidate-boot validation (packer-gcp.yml calls it)
└── release-gcp.yml # GCP release images and deployer image (release-please.yml calls it)
Underscore prefix (_*.yml) indicates reusable workflows called by deploy.yml.
Dependency Chain¶
Quality (must pass first)
│
▼
Core (ECR)
│
├──────────────┐
▼ ▼
Range Shifter Engine
│ │
└──────────────┘
│
▼
Shifter Platform Plan/Deploy
Downstream AWS jobs use explicit needs.<job>.result gates. A skipped upstream
is acceptable when its path filter did not select it, but a failed or cancelled
upstream must stop dependent infrastructure work. In particular, BigRAE Platform
must not plan or apply on top of a failed BigRAE Engine deploy.
Change Detection¶
The orchestrator uses path filters to run only relevant jobs:
| Filter | Triggers When |
|---|---|
core |
ECR module, environment root, deploy workflow |
range |
Range Terraform, engine state module |
shifter_engine |
BigRAE Engine code, ECR module |
shifter_platform |
Portal/Guacamole Terraform and platform deploy workflow |
quality_relevant |
Any non-docs change, plus guardrail docs; controls whether Quality runs without launching deploy or Terraform plan jobs |
portal_image |
Portal image build inputs (shifter/shifter_platform/**, cyberscript, installation, .dockerignore); triggers the portal image build/deploy on a deploy dispatch without running Terraform |
gcp |
GCP Terraform, the Helm chart, GCP scripts, the GCP deploy command modules in scripts/bootstrap, GCP cloud adapters |
mcp |
MCP package changes, routed to Quality only |
quality_only |
Non-deploy test-support and guardrail surfaces (scripts/stack-smoke/**, each development pack under shifter/environment-packs/, _quality.yml, ADR/guardrail checker paths), routed to Quality only. Shipped environment packs route to portal_image |
Quality Gate¶
Runs on every PR, direct push to dev/main, and every manual deploy dispatch
(the safety gate before an apply) unless the diff is ordinary docs-only.
Guardrail docs such as .github/pull_request_template.md,
.github/copilot-instructions.md, docs/adr/**, and the ADR enforcement
guide are quality-relevant and still run Quality.
The deploy workflow exposes one quality_relevant output for this decision:
non-docs changes make it true, ordinary docs-only changes make it false, and
guardrail docs make it true even though they are documentation. PR Gate accepts
a skipped Quality job only when quality_relevant is false.
Inside _quality.yml, the always-run path-detection job runs
scripts/quality_ownership/classify_paths.py, which reads the versioned
quality-ownership contract at .github/quality-path-filters.yaml and rejects
any unclassified changed path before it selects jobs. That contract also maps
every production path to its blocking lint, security, and test jobs; the
quality-path-ownership ADR guard (ADR-004-R24) enforces that ownership.
See docs/technical/dev/adr-enforcement.md for how to add a quality unit or
record a gap.
The blocking repository-wide owners include:
- ADR tooling: Ruff plus medium-and-higher Bandit findings.
- Stack smoke: Ruff, medium-and-higher Bandit findings, and ShellCheck.
- Standalone deployment utilities: one blocking medium-and-higher Bandit lane in addition to their package-local Ruff and test jobs.
- MCP packages: ESLint security rules promoted to errors, zero warning
tolerance, package tests (including
mcp/shared), and dependency audits. - Container inputs: BuildKit Dockerfile checks, entrypoint ShellCheck, and
high/critical Trivy configuration policy; the Guacamole authority test job
builds both customized runtime images. The root
.dockerignoreis a separate portal-image-context owner whose test route builds and boots the production portal image through stack smoke. - Helm charts: lint and render every supported values profile, then scan the rendered manifests with kube-linter.
The BigRAE Engine reusable workflow has an additional blocking provisioner pytest gate before local Docker validation, credentialed image build, and ECS deploy. This keeps a deploy dispatch from building or deploying the provisioner image when the top-level Quality job is legitimately skipped.
- ADR conformance:
python3 scripts/adr_guard/adr_guard.py --all --level ciIncludesadr-registry,layer-imports,cross-layer-model-imports, andcloud-factory-seam(ADR-005-R1 cloud adapter parity). - Workflow linting:
actionlint - Terraform linting:
tflintwithtflint-ruleset-googleplugin The repo currently runs a narrow TFLint profile that excludes existing version/provider and unused-declaration debt until that backlog is burned down. The Google plugin adds GCP-specific rules (invalid machine types, deprecated attributes, etc.). - Python import contracts:
lint-imports --config ../../.importlinter - Python linting:
ruff check,ruff format --check - K8s schema validation:
kubeconformvalidates Kubernetes manifests against official schemas, pinned to the target GKE version. - K8s security and best practices:
kube-linterenforces security contexts, resource limits, privilege escalation prevention, and other best practices via.kube-linter.yaml. - K8s security scanning: Checkov with the
kubernetesframework. Current soft-fail is scoped to Kubernetes manifest hardening and does not justify Terraform soft-fail. - Tests: package-local Python, JavaScript, and harness suites, including
shifter_platform,cyberscript,shifter/engine/provisioner,packer,installation,scripts/bootstrap,scripts/gcp,scripts/stack-smoke, the Postgres migration proof, and MCP package tests includingmcp/planner. - IaC scanning: Checkov for Terraform is a blocking gate under
ADR-004-R11. Pre-commit and CI share the same config at
platform/terraform/.checkov.yaml;--soft-failis off. Accepted-risk waivers (Checkovskip-checkentries or inline# checkov:skip=…comments) require a matching entry indocs/adr/exceptions.yamlwith owner, reason, expiry, affected paths, and the Checkov policy ID. - Secret scanning: gitleaks on newly introduced commits
- Coverage:
shifter_platformand provisioner test jobs emit XML coverage reports
Commit-message or label-based test skips are not accepted by deploy.yml.
Terraform Flow¶
Each component follows the same pattern:
- Plan job:
- Checkout repo (tfvars are committed)
terraform initterraform validateterraform plan -lock-timeout=5m -out=tfplan-
Comment plan on PR (if PR)
-
Apply job (if plan succeeds):
- Skip on PRs
- Create a local saved
tfplanwithterraform plan -lock-timeout=5m -out=tfplan terraform apply -lock-timeout=5m tfplan
Branch and manual deploy runs are queued by the Deploy workflow's concurrency group so a newer push cannot cancel a Terraform process after it has started mutating remote infrastructure or while it holds the backend lock. Pull request runs may still be cancelled by newer commits because they do not execute apply jobs.
The saved plan file created inside the apply job is the apply contract. If state
moves after that local plan, Terraform should fail the saved-plan apply instead
of silently executing a fresh unplanned apply. Raw binary plans are not uploaded
as workflow artifacts because they can include unredacted plan/state data. The
platform Service Discovery replacement check reads the same saved tfplan that
the apply step consumes.
Note: The committed terraform.tfvars files ship an example.com
baseline. Deployment-specific values (domains, alarm emails, allow-list
domains, account-suffixed bucket names, GCP project id, etc.) come from
GitHub repository variables and secrets at deploy time; CI/CD renders
them into a gitignored local.auto.tfvars before terraform apply.
See docs/dev/deploy-secrets.md
for the required surface.
AWS Platform Deployment¶
The Core -> Range -> Engine -> Platform chain above is the order between the
reusable workflows. This section documents the ordered sequence of jobs and
steps inside the platform reusable workflow (_shifter-platform.yml), which
is the portal (platform) stack. Jobs run in this order (each needs the
previous):
push-guacamole-images: build and push theguacdandguacamole-clientimages to ECR (only when the pinned tag is absent). Runs first because the portal plan and apply reference them.plan: renderlocal.auto.tfvarsfromTF_VARS_<ENV>_PORTAL(and the engine image digest intozz-engine-image.auto.tfvars), render the backend config, theninit/validate/plan.apply(branch/dispatch only), in order:- Drain Service Discovery registrations the plan will delete (scale affected ECS services to zero) so replacements do not collide.
terraform applythe saved plan.- Restore ECS desired counts after apply.
- Assert portal inspection route and endpoint wiring (no-op when inspection is off; fails the deploy if the routed firewall path is unhealthy).
- Verify RDS pending modifications applied (dev only): a successful apply
that leaves non-empty
PendingModifiedValuesfails the job. - Wait for Guacamole ECS services to stabilize (bounded poll).
build: build and push the portal image (shifter/shifter_platform/Dockerfile) toshifter-<env>-portal. The image is tagged<short-sha>-<run-id>-<run-attempt>and the job outputs its digest; there is nolatesttag.deploy, in order:- Resolve topology from Terraform state (
enable_autoscaling,ec2_instance_id,asg_name). - Update Parameter Store with the new image digest and tag.
- Update bootstrap admin parameters (staff / superuser emails).
- Run database migrations on one healthy instance (ASG mode; single instance mode migrates inside its deploy script).
- Deploy via SSM (single instance) or trigger an ASG instance refresh (ASG mode).
- Verify ASG image digest and verify ASG worker health (ASG mode).
verify: post-deploy health check.post-deploy-smoke(dev applies only,continue-on-error): runscripts/smoke-test.sh. A failure opens a[smoke-test]bug issue and does not fail the deploy. Requires theSMOKE_*secrets (seedocs/dev/deploy-secrets.md).
Single instance mode targets the {env}-portal-ec2 tagged instance.
Auto Scaling mode refreshes {env}-portal-asg and verifies the new digest
and worker health on every in-service instance.
Refresh scope and timing (issue #1639). The ASG instance refresh runs only when the
portal actually changed (the portal image or platform Terraform). An engine-provisioner-only
deploy no longer triggers a full portal refresh. Refresh readiness tracks the ALB target
group (health_check_type = "ELB"), so a refresh converges once the portal is actually
serving instead of waiting out the EC2 "insufficient data to evaluate its health" window.
The ASG timing is environment-owned Terraform (health_check_type,
health_check_grace_period, and instance_refresh_instance_warmup on the portal root),
so a dev or proof tenant can shorten the health-grace to speed up the deploy, test, fix
loop without changing prod.
First-run DNS timing¶
On a fresh account the first Portal apply blocks while AWS validates ACM
certificates and SES identities. Publish the validation records in the
authoritative DNS zone while the apply is waiting, and publish the runtime
routing records (ALB CNAMEs, CTFd A record) once the apply creates those
endpoints. The exact records and commands are in
docs/dev/deploy-secrets.md
under the first-run DNS validation and routing sections.
Environment Detection¶
Event / input → Behavior
Pull request → Quality only
Push to dev / main → Quality only
dispatch environment=aws-dev → AWS dev deploy
Provider Routing¶
dev/mainare integration branches: they run the quality gate but never plan/apply infrastructure or deploy workloads.- A deploy is a manual
workflow_dispatchthat names theenvironment; the branch you run it from is the code that deploys. No branch name selects the target. - The GCP control plane is deployed through the Helm chart in
platform/charts/shifterbyscripts/bootstrap/deploy.py gcp-deploy, with generated values layered on top of the profile values. CI and an operator without GitHub run the same command (ADR-070). - The GCP portal auth contract is FirebaseUI/browser-side Identity Platform auth plus server-side verified-token exchange. Do not add Django credential handling to recreate Cognito semantics.
- Multi-cloud work enters through the shared cloud adapter layers rather than provider-specific calls in domain services.
Manual Deploy Inputs¶
deploy.yml runs on workflow_dispatch. The environment input selects the deployment target; the bootstrap flags are strict by default.
environment: which environment to deploy:aws-devorgcp-dev. A closed choice enum. The branch you run the workflow from is the code that deploys.aws_first_deploy(defaultfalse): allow the AWS engine deploy to skip the ECS task-family existence check. Settrueonly for the first-ever deploy to a fresh AWS environment, before the platform Terraform apply has created the provisioner task definition. On any normal deploy a missing or typo'd task family fails the run instead of skipping silently. Clear it (re-run without the flag) once the platform stack has been applied.gcp_require_active_certificate(defaulttrue): require the GKE ManagedCertificate to be Active for the public hostname. Setfalseonly for first-time GCP bootstrap, before DNS for the hostname has been pointed at the ingress IP.
Runner Policy¶
Workflows choose one runner class per job. Portable quality jobs run on
ubuntu-latest; trusted deployment, image-build, Packer, and
environment-mutating jobs run on self-hosted runners. GitHub Actions does not
provide native fallback between those runner classes.
The self-hosted runner has:
- AWS CLI configured
- gcloud SDK support for GCP workflows
- Docker + BuildX
- Terraform (AWS deploy jobs pin
1.13.3; GCP deploy jobs pin1.17.1) - Python 3.12
- Network access to AWS and GCP APIs
Viewing Logs¶
- Go to Actions tab in GitHub
- Select the workflow run
- Expand the job you want to inspect
- Each step shows its logs
Terraform plans are also posted as PR comments.
Common Issues¶
Workflow Doesn't Trigger¶
- Check branch protection rules
- Verify path filters match your changes
- Look for
paths-filterin deploy.yml - A deploy is a manual dispatch:
gh workflow run deploy.yml --ref <branch> -f environment=aws-dev. Pushes and PRs run validation only; no branch push deploys.
Terraform Plan Fails¶
- Check for formatting issues:
terraform fmt -recursive - Validate locally first:
terraform validate - Review the error in the Actions log
Docker Build Fails¶
- Check Dockerfile syntax
- Verify base image availability
- Review build logs for dependency issues
Deploy Fails¶
- Check EC2 instance is running
- Verify SSM agent is healthy
- Review SSM command output in AWS console
GCP Deploy Fails¶
- Verify each purpose Environment has its explicit
GCP_*_SERVICE_ACCOUNTandGCP_WORKLOAD_IDENTITY_PROVIDERsecrets set - Check the
Render deployment inputsstep: the CI adapter stops when a required input is missing or when a GitHub value conflicts with the root config - Check the
Run the infrastructure stagestep for IAM or bucket-name conflicts on the GCS backend bucket. The step log shows each command that ran and the error of the command that failed - The command keeps the Terraform outputs, the runtime env, and the generated Helm values in a private temporary directory, not in the workflow log. Review the ingress resources and the
BackendConfigresources if hostname, DNS, certificate, or Cloud Armor behavior is wrong - Review the
guacamole-runtimeSecret sync in theRun the rollout stagestep if the Guacamole client pods stay inCreateContainerConfigError - If the portal auth path is wrong, verify the Terraform outputs expose
public_hostname,managed_tls_enabled=true, that Identity Platform was provisioned successfully, that the blocking function and MFA configuration are present, that the bootstrap operator credentials were supplied, and that the managed certificate reachesActive - Check
kubectl rollout statusoutput for the specific control-plane deployment that stalled