Terraform¶
Infrastructure layout and local Terraform commands.
Directory Structure¶
platform/terraform/
├── environments/
│ ├── dev/
│ │ ├── main.tf # ECR repositories
│ │ ├── backend.tf # S3 state config
│ │ ├── variables.tf
│ │ ├── terraform.tfvars # Environment config (committed)
│ │ ├── portal/ # Portal infrastructure
│ │ │ ├── main.tf
│ │ │ ├── backend.tf
│ │ │ └── terraform.tfvars
│ │ └── range/ # Range VPC infrastructure
│ │ ├── main.tf
│ │ ├── backend.tf
│ │ └── terraform.tfvars
│ ├── prod/ # Same structure as dev
│ └── proof/ # Same structure as dev
├── modules/ # Reusable modules
│ ├── portal/
│ │ ├── alb/ # Application Load Balancer, WAF
│ │ ├── backup-alerts/ # Backup failure alerting
│ │ ├── cognito/ # User pool, OIDC
│ │ ├── ctfd/ # CTFd host and networking
│ │ ├── ec2/ # Portal instances
│ │ ├── messaging/ # SNS/SQS for events
│ │ ├── rds/ # PostgreSQL database
│ │ ├── redis/ # ElastiCache Redis
│ │ ├── s3/ # User storage bucket
│ │ ├── ses/ # Transactional email identities
│ │ ├── ssm/ # SSM parameters
│ │ └── vpc/ # Portal VPC, subnets, inspection firewall
│ ├── range/
│ │ └── vpc/ # Range VPC, Network Firewall
│ ├── engine-provisioner/ # ECS task for Shifter Engine
│ ├── engine-state/ # Engine state bucket and DynamoDB runtime locks
│ ├── guacamole/ # Browser-based RDP
│ ├── log-aggregation/ # Centralized logging
│ └── ecr/ # Container registries
└── global/ # Cross-environment resources
├── iam/ # GitHub OIDC, roles
├── github-runner/ # Self-hosted runners (+ runner-health)
├── tssummit/ # Technical Summit event stack
├── tssummit-ranges/ # Technical Summit event ranges
└── dev-box/ # Windows dev instance
GCP work lives in a parallel tree so it cannot collide with the AWS deploy path:
platform/terraform/gcp/
├── environments/
│ └── gcp-dev/ # GCP control-plane foundation for gcp-dev
└── modules/
└── platform-core/ # VPC, GKE, Cloud SQL, Memorystore, Artifact Registry, Pub/Sub, GCS, ingress IP, Cloud Armor, optional DNS, secrets
The GCP control plane is packaged separately as a Helm chart:
platform/charts/shifter/
├── templates/ # Deployments, services, ingress, BackendConfig, RBAC
├── values.yaml # Chart defaults
├── values-gcp-dev.yaml # gcp-dev environment overrides
└── values-gcp-prod.yaml # gcp-prod environment overrides
State Management¶
Each component has its own state file:
| Component | State Key |
|---|---|
| Core (ECR) | shifter/{env}/terraform.tfstate |
| Range | {env}/range/terraform.tfstate |
| Portal | {env}/portal/terraform.tfstate |
| Global | global/{component}/terraform.tfstate |
Only the Core key carries the shifter/ prefix; the Range and Portal keys do
not. The Portal stack reads the Core and Range state as remote-state data
sources, so these keys must match exactly. For the full stack apply order,
per-stack backend keys, and the "every value you must override before first
apply" checklist, see
docs/dev/aws-terraform-apply-order.md.
AWS Terraform state is stored in S3 with S3 native locking
(use_lockfile = true):
- Dev: shifter-dev-infra-* bucket
- Prod: shifter-infra-* bucket
The engine-state module creates a DynamoDB table for provisioner runtime
locks. It is not the Terraform backend lock.
Working Locally¶
Initialize¶
For the staged GCP tree, local validation still runs without a remote backend:
cd platform/terraform/gcp/environments/platform
terraform init -backend=false -lockfile=readonly
terraform validate
scripts/bootstrap/deploy.py gcp-deploy creates and hardens the GCS backend
bucket ${project_id}-terraform-state before it runs terraform init, for an
operator and for CI. A deployment that runs without GitHub keeps its Terraform
values in its deployment input bundle. The committed gcp-dev tfvars file
carries the public edge and operator-access settings of that CI-deployed
environment:
public_hostname = "shifter.example.com"
enable_managed_tls = true
gke_master_authorized_cidrs = []
create_dns_managed_zone = false
dns_managed_zone_name = ""
dns_zone_dns_name = ""
dns_record_ttl = 300
gcp-deploy sets public_hostname from deployment.domain and sets
enable_managed_tls = true; those values win over a tfvars value. It fails
before Terraform apply unless gke_master_authorized_cidrs is empty or holds
only connected RFC1918 networks. Applies of the GCP platform root run through gcp-deploy.
Plan¶
Apply (Dev Only)¶
Never apply to prod locally - use the CI/CD pipeline.
RDS Change Application¶
RDS changes that AWS can apply during a deploy (instance class, storage
size, engine version, and dynamic parameter-group fields) must not be
treated as complete until AWS reports no pending modifications for the
affected instance. For dev-only RDS resources, prefer explicit
apply_immediately module inputs that default to applying intended changes
during the current deploy. Production must keep change timing deliberate:
either pass an explicit maintenance-window choice through the same module
surface or use a separate reviewed change flow.
apply_immediately does NOT remediate every RDS change. Static
parameter-group fields and major version upgrades still require an instance
reboot to take effect, and modify-class operations can still leave
PendingModifiedValues populated until the underlying maintenance action
completes. The post-apply check will surface that residual state, but it does
not by itself drive a reboot. When you change one of those fields, plan a
follow-up reboot (or a maintenance-window-gated path) explicitly.
Post-apply checks belong at the Terraform workflow boundary, after
terraform apply, where the existing AWS credentials and environment context
are already present. Reuse Terraform outputs or module outputs to identify
the managed DB instances; do not hardcode names such as dev-portal-db
inside a generic checker. For dev portal deploys, a successful apply that
leaves non-empty PendingModifiedValues is treated as an incomplete deploy
and fails the job loudly. Prod intentionally skips the check because prod
relies on the maintenance-window path.
Configuration vs Secrets¶
In terraform.tfvars (committed baseline)¶
# Non-deployment-specific environment config - committed.
aws_region = "us-east-2"
environment = "dev"
instance_type = "t3.large"
domain_name = "dev.shifter.example.com" # example.com baseline
enable_autoscaling = false
In local.auto.tfvars (gitignored, per-deployment override)¶
# Deployment-specific identifiers - never committed.
domain_name = "dev.shifter.your-domain.example"
ses_domain = "your-domain.example"
alarm_email = "your-team@your-domain.example"
allowed_email_domains = ["your-domain.example"]
user_storage_bucket = "shifter-dev-user-storage-<account-id>"
Terraform auto-loads *.auto.tfvars files alongside terraform.tfvars
and the .local/.auto.tfvars values win. AWS portal CI deploy workflows
render local.auto.tfvars from GitHub secrets; see
docs/dev/deploy-secrets.md.
The AWS portal deploy job derives its single-instance vs. ASG path from the
applied Terraform state, not from a GitHub Actions variable. The portal roots
export enable_autoscaling, ec2_instance_id, and asg_name; the deploy
workflow reads those outputs before updating the portal image. In
single-instance mode it also verifies that exactly one running EC2 instance is
tagged for the portal before sending the SSM deploy command. In ASG mode it
refreshes the Terraform-named ASG and verifies the new image tag on every
in-service instance after the refresh.
In Cloud Secret Managers (Runtime)¶
Database passwords, API keys, and runtime signing keys are stored in the cloud
secret manager and accessed at runtime. For the current gcp-dev slice,
Terraform seeds the app, DB, and Guacamole runtime bundles needed for first
boot. GCP no longer relies on a separately managed OIDC runtime secret; it
provisions Identity Platform directly and bootstrap seeds the first operator
account for the secure portal path. The secure portal flow uses browser-side
Google auth and app-side verified-token exchange rather than server-side
password handling in Django.
Module Patterns¶
Input Validation¶
variable "environment" {
type = string
validation {
condition = contains(["dev", "prod"], var.environment)
error_message = "Environment must be dev or prod."
}
}
Consistent Tagging¶
locals {
common_tags = {
Project = "shifter"
Environment = var.environment
ManagedBy = "terraform"
}
}
resource "aws_instance" "example" {
# ...
tags = merge(local.common_tags, {
Name = "${var.environment}-example"
})
}
Resource Naming¶
Pattern: shifter-{environment}-{component}
Examples:
- shifter-dev-portal-ec2
- shifter-prod-range-vpc
- shifter-dev-user-storage
Provider Configuration¶
terraform {
required_version = ">= 1.5.0"
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 6.0"
}
}
}
provider "aws" {
region = var.aws_region
default_tags {
tags = {
Project = "shifter"
Environment = var.environment
ManagedBy = "terraform"
}
}
}
Common Operations¶
Import Existing Resource¶
Remove Resource from State¶
View Current State¶
AWS_PROFILE=$PANW_SHIFTER_DEV_PROFILE terraform state list
AWS_PROFILE=$PANW_SHIFTER_DEV_PROFILE terraform state show aws_instance.example
Format All Files¶
Adding New Infrastructure¶
- Create module in
platform/terraform/modules/if reusable - Add to environment in
platform/terraform/environments/{env}/ - Update tfvars with required variables
- Test in dev before prod
For GCP work:
- Add reusable modules under
platform/terraform/gcp/modules/ - Wire them into
platform/terraform/gcp/environments/platform/ - Provision provider-native shared services there first: Pub/Sub, GCS, Secret Manager, Cloud SQL, Memorystore, ingress IPs
- Stage the matching chart changes under
platform/charts/shifter/; the chart is the only GCP Kubernetes renderer - Keep the remote backend, deploy credentials, and rendered runtime config aligned with the Terraform outputs
- Do not weaken or replace the existing AWS environment trees while expanding GCP
Debugging¶
Enable Logging¶
Common Issues¶
State Lock Error
Provider Cache Issues
Plan Drift
- Someone made manual changes in console
- Run terraform plan to see differences
- Either import or recreate the resource
Don'ts¶
- Don't use
terraform apply -auto-approvelocally - Don't hardcode values that vary by environment
- Don't store state locally (use the existing remote backend for the target environment)
- Don't skip
terraform planreview - Don't apply prod changes without PR review