Skip to content

Terraform

Infrastructure layout and local Terraform commands.

Directory Structure

platform/terraform/
├── environments/
│   ├── dev/
│   │   ├── main.tf           # ECR repositories
│   │   ├── backend.tf        # S3 state config
│   │   ├── variables.tf
│   │   ├── terraform.tfvars  # Environment config (committed)
│   │   ├── portal/           # Portal infrastructure
│   │   │   ├── main.tf
│   │   │   ├── backend.tf
│   │   │   └── terraform.tfvars
│   │   └── range/            # Range VPC infrastructure
│   │       ├── main.tf
│   │       ├── backend.tf
│   │       └── terraform.tfvars
│   ├── prod/                 # Same structure as dev
│   └── proof/                # Same structure as dev
├── modules/                  # Reusable modules
│   ├── portal/
│   │   ├── alb/              # Application Load Balancer, WAF
│   │   ├── backup-alerts/    # Backup failure alerting
│   │   ├── cognito/          # User pool, OIDC
│   │   ├── ctfd/             # CTFd host and networking
│   │   ├── ec2/              # Portal instances
│   │   ├── messaging/        # SNS/SQS for events
│   │   ├── rds/              # PostgreSQL database
│   │   ├── redis/            # ElastiCache Redis
│   │   ├── s3/               # User storage bucket
│   │   ├── ses/              # Transactional email identities
│   │   ├── ssm/              # SSM parameters
│   │   └── vpc/              # Portal VPC, subnets, inspection firewall
│   ├── range/
│   │   └── vpc/              # Range VPC, Network Firewall
│   ├── engine-provisioner/   # ECS task for Shifter Engine
│   ├── engine-state/         # Engine state bucket and DynamoDB runtime locks
│   ├── guacamole/            # Browser-based RDP
│   ├── log-aggregation/      # Centralized logging
│   └── ecr/                  # Container registries
└── global/                   # Cross-environment resources
    ├── iam/                  # GitHub OIDC, roles
    ├── github-runner/        # Self-hosted runners (+ runner-health)
    ├── tssummit/             # Technical Summit event stack
    ├── tssummit-ranges/      # Technical Summit event ranges
    └── dev-box/              # Windows dev instance

GCP work lives in a parallel tree so it cannot collide with the AWS deploy path:

platform/terraform/gcp/
├── environments/
│   └── gcp-dev/             # GCP control-plane foundation for gcp-dev
└── modules/
    └── platform-core/       # VPC, GKE, Cloud SQL, Memorystore, Artifact Registry, Pub/Sub, GCS, ingress IP, Cloud Armor, optional DNS, secrets

The GCP control plane is packaged separately as a Helm chart:

platform/charts/shifter/
├── templates/              # Deployments, services, ingress, BackendConfig, RBAC
├── values.yaml             # Chart defaults
├── values-gcp-dev.yaml     # gcp-dev environment overrides
└── values-gcp-prod.yaml    # gcp-prod environment overrides

State Management

Each component has its own state file:

Component State Key
Core (ECR) shifter/{env}/terraform.tfstate
Range {env}/range/terraform.tfstate
Portal {env}/portal/terraform.tfstate
Global global/{component}/terraform.tfstate

Only the Core key carries the shifter/ prefix; the Range and Portal keys do not. The Portal stack reads the Core and Range state as remote-state data sources, so these keys must match exactly. For the full stack apply order, per-stack backend keys, and the "every value you must override before first apply" checklist, see docs/dev/aws-terraform-apply-order.md.

AWS Terraform state is stored in S3 with S3 native locking (use_lockfile = true): - Dev: shifter-dev-infra-* bucket - Prod: shifter-infra-* bucket

The engine-state module creates a DynamoDB table for provisioner runtime locks. It is not the Terraform backend lock.

Working Locally

Initialize

cd platform/terraform/environments/dev/portal
AWS_PROFILE=$PANW_SHIFTER_DEV_PROFILE terraform init

For the staged GCP tree, local validation still runs without a remote backend:

cd platform/terraform/gcp/environments/platform
terraform init -backend=false -lockfile=readonly
terraform validate

scripts/bootstrap/deploy.py gcp-deploy creates and hardens the GCS backend bucket ${project_id}-terraform-state before it runs terraform init, for an operator and for CI. A deployment that runs without GitHub keeps its Terraform values in its deployment input bundle. The committed gcp-dev tfvars file carries the public edge and operator-access settings of that CI-deployed environment:

public_hostname             = "shifter.example.com"
enable_managed_tls          = true
gke_master_authorized_cidrs = []
create_dns_managed_zone = false
dns_managed_zone_name   = ""
dns_zone_dns_name       = ""
dns_record_ttl          = 300

gcp-deploy sets public_hostname from deployment.domain and sets enable_managed_tls = true; those values win over a tfvars value. It fails before Terraform apply unless gke_master_authorized_cidrs is empty or holds only connected RFC1918 networks. Applies of the GCP platform root run through gcp-deploy.

Plan

AWS_PROFILE=$PANW_SHIFTER_DEV_PROFILE terraform plan

Apply (Dev Only)

AWS_PROFILE=$PANW_SHIFTER_DEV_PROFILE terraform apply

Never apply to prod locally - use the CI/CD pipeline.

RDS Change Application

RDS changes that AWS can apply during a deploy (instance class, storage size, engine version, and dynamic parameter-group fields) must not be treated as complete until AWS reports no pending modifications for the affected instance. For dev-only RDS resources, prefer explicit apply_immediately module inputs that default to applying intended changes during the current deploy. Production must keep change timing deliberate: either pass an explicit maintenance-window choice through the same module surface or use a separate reviewed change flow.

apply_immediately does NOT remediate every RDS change. Static parameter-group fields and major version upgrades still require an instance reboot to take effect, and modify-class operations can still leave PendingModifiedValues populated until the underlying maintenance action completes. The post-apply check will surface that residual state, but it does not by itself drive a reboot. When you change one of those fields, plan a follow-up reboot (or a maintenance-window-gated path) explicitly.

Post-apply checks belong at the Terraform workflow boundary, after terraform apply, where the existing AWS credentials and environment context are already present. Reuse Terraform outputs or module outputs to identify the managed DB instances; do not hardcode names such as dev-portal-db inside a generic checker. For dev portal deploys, a successful apply that leaves non-empty PendingModifiedValues is treated as an incomplete deploy and fails the job loudly. Prod intentionally skips the check because prod relies on the maintenance-window path.

Configuration vs Secrets

In terraform.tfvars (committed baseline)

# Non-deployment-specific environment config - committed.
aws_region         = "us-east-2"
environment        = "dev"
instance_type      = "t3.large"
domain_name        = "dev.shifter.example.com"  # example.com baseline
enable_autoscaling = false

In local.auto.tfvars (gitignored, per-deployment override)

# Deployment-specific identifiers - never committed.
domain_name           = "dev.shifter.your-domain.example"
ses_domain            = "your-domain.example"
alarm_email           = "your-team@your-domain.example"
allowed_email_domains = ["your-domain.example"]
user_storage_bucket   = "shifter-dev-user-storage-<account-id>"

Terraform auto-loads *.auto.tfvars files alongside terraform.tfvars and the .local/.auto.tfvars values win. AWS portal CI deploy workflows render local.auto.tfvars from GitHub secrets; see docs/dev/deploy-secrets.md.

The AWS portal deploy job derives its single-instance vs. ASG path from the applied Terraform state, not from a GitHub Actions variable. The portal roots export enable_autoscaling, ec2_instance_id, and asg_name; the deploy workflow reads those outputs before updating the portal image. In single-instance mode it also verifies that exactly one running EC2 instance is tagged for the portal before sending the SSM deploy command. In ASG mode it refreshes the Terraform-named ASG and verifies the new image tag on every in-service instance after the refresh.

In Cloud Secret Managers (Runtime)

Database passwords, API keys, and runtime signing keys are stored in the cloud secret manager and accessed at runtime. For the current gcp-dev slice, Terraform seeds the app, DB, and Guacamole runtime bundles needed for first boot. GCP no longer relies on a separately managed OIDC runtime secret; it provisions Identity Platform directly and bootstrap seeds the first operator account for the secure portal path. The secure portal flow uses browser-side Google auth and app-side verified-token exchange rather than server-side password handling in Django.

Module Patterns

Input Validation

variable "environment" {
  type = string
  validation {
    condition     = contains(["dev", "prod"], var.environment)
    error_message = "Environment must be dev or prod."
  }
}

Consistent Tagging

locals {
  common_tags = {
    Project     = "shifter"
    Environment = var.environment
    ManagedBy   = "terraform"
  }
}

resource "aws_instance" "example" {
  # ...
  tags = merge(local.common_tags, {
    Name = "${var.environment}-example"
  })
}

Resource Naming

Pattern: shifter-{environment}-{component}

Examples: - shifter-dev-portal-ec2 - shifter-prod-range-vpc - shifter-dev-user-storage

Provider Configuration

terraform {
  required_version = ">= 1.5.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
  }
}

provider "aws" {
  region = var.aws_region

  default_tags {
    tags = {
      Project     = "shifter"
      Environment = var.environment
      ManagedBy   = "terraform"
    }
  }
}

Common Operations

Import Existing Resource

AWS_PROFILE=$PANW_SHIFTER_DEV_PROFILE terraform import \
  aws_instance.example i-1234567890abcdef0

Remove Resource from State

AWS_PROFILE=$PANW_SHIFTER_DEV_PROFILE terraform state rm \
  aws_instance.example

View Current State

AWS_PROFILE=$PANW_SHIFTER_DEV_PROFILE terraform state list
AWS_PROFILE=$PANW_SHIFTER_DEV_PROFILE terraform state show aws_instance.example

Format All Files

terraform fmt -recursive

Adding New Infrastructure

  1. Create module in platform/terraform/modules/ if reusable
  2. Add to environment in platform/terraform/environments/{env}/
  3. Update tfvars with required variables
  4. Test in dev before prod

For GCP work:

  1. Add reusable modules under platform/terraform/gcp/modules/
  2. Wire them into platform/terraform/gcp/environments/platform/
  3. Provision provider-native shared services there first: Pub/Sub, GCS, Secret Manager, Cloud SQL, Memorystore, ingress IPs
  4. Stage the matching chart changes under platform/charts/shifter/; the chart is the only GCP Kubernetes renderer
  5. Keep the remote backend, deploy credentials, and rendered runtime config aligned with the Terraform outputs
  6. Do not weaken or replace the existing AWS environment trees while expanding GCP

Debugging

Enable Logging

export TF_LOG=DEBUG
terraform plan

Common Issues

State Lock Error

# Force unlock (use carefully)
terraform force-unlock LOCK_ID

Provider Cache Issues

rm -rf .terraform
terraform init

Plan Drift - Someone made manual changes in console - Run terraform plan to see differences - Either import or recreate the resource

Don'ts

  • Don't use terraform apply -auto-approve locally
  • Don't hardcode values that vary by environment
  • Don't store state locally (use the existing remote backend for the target environment)
  • Don't skip terraform plan review
  • Don't apply prod changes without PR review