GCP Infrastructure¶
GKE-based deployment of the BigRAE platform on Google Cloud.
Directory Structure¶
platform/
├── terraform/gcp/
│ ├── modules/platform-core/ # All GCP platform infrastructure
│ ├── environments/platform/ # Shared platform root for every deployment
│ ├── environments/<deployment>/ # Committed values of the CI-deployed tenants
│ └── global/ # cicd-oidc and github-runner roots
├── charts/shifter/ # Helm chart: the only GCP Kubernetes renderer
└── deployer/gcp/ # Deployer image of one published release
Terraform provisions the GCP resources. The Helm chart packages the BigRAE
control plane. The deploy command (scripts/bootstrap/deploy.py gcp-deploy)
renders the generated values from the Terraform outputs, the root config, and
the Secret Manager payloads. Then it installs the chart. A deployment that runs
without GitHub keeps its own Terraform values in its deployment input bundle,
not in this repository.
Terraform Module: platform-core¶
Single module provisions the entire GCP control plane.
| Resource | Service | Purpose |
|---|---|---|
| VPC Networks (×2) | VPC | platform (GKE, shared services) and range (guest isolation) |
| GKE Cluster | GKE | Private nodes, VPC-native, Workload Identity enabled, Binary Authorization enforced via project singleton policy, public control-plane endpoint restricted by authorized CIDRs |
| Node Pools (×3) | GKE | web (portal, Guacamole), workers (domain workers), provisioner (range provisioning jobs) |
| Cloud SQL | Cloud SQL | PostgreSQL. Hosts platform DB and Guacamole DB. Private IP only. |
| Memorystore | Memorystore | Redis. Channel layer and worker coordination. |
| Pub/Sub | Pub/Sub | Event topic with per-domain subscriptions (cms, engine, mc). |
| Artifact Registry | Artifact Registry | Container repositories (portal, guacd, guacamole-client, pulumi-provisioner). |
| Secret Manager | Secret Manager | Runtime secret bundles (app, db, guacamole-db, oidc, guacamole-json-auth). |
| Cloud DNS | Cloud DNS | Optional. Public hostname with Google-managed TLS certificate. |
| Cloud Armor | Cloud Armor | Baseline WAF policy attached to the public portal and Guacamole GKE backends. |
| GCS Buckets | Cloud Storage | Assets/artifacts and Terraform state. Each Terraform-owned bucket uses its own customer-managed Cloud KMS key (Bucket encryption keys). |
| Service Accounts | IAM | Separate accounts for GKE nodes and workloads (portal, workers, provisioner). Workload Identity binds K8s SAs to GCP SAs. |
GKE Cluster¶
Private nodes with no public node IPs. Cloud NAT provides outbound connectivity.
- Release channel: REGULAR (automatic patching)
- Workload Identity: Enabled. Pod service accounts map to GCP service accounts.
- Networking: VPC-native with secondary IP ranges for pods and services
- Logging/Monitoring: System components and workload logging enabled
- Control-plane access: private endpoint reached through Connect Gateway; optional direct access accepts connected RFC1918 networks only
- Binary Authorization: enabled with
PROJECT_SINGLETON_POLICY_ENFORCE
Node Pools¶
| Pool | Role | Notes |
|---|---|---|
web |
Portal and Guacamole | Handles user-facing traffic |
workers |
CMS, Engine, MC background workers | Queue consumers |
provisioner |
Range provisioning jobs | Runs K8s Jobs for range lifecycle |
Machine types and node counts are configurable per environment.
Kubernetes Workloads¶
Managed via Helm with the chart defaults, the profile values, and the values that gcp-deploy generates.
Chart Resources (platform/charts/shifter/templates/)¶
| Manifest | Workload |
|---|---|
web-deployment.yaml |
Portal Django app |
guacd-deployment.yaml |
Guacamole protocol daemon |
guacamole-client-deployment.yaml |
Guacamole web client |
worker-cms-deployment.yaml |
CMS queue consumer |
worker-engine-deployment.yaml |
Engine queue consumer |
worker-mc-deployment.yaml |
Mission Control queue consumer |
ctf-scheduler-deployment.yaml |
CTF batch scheduler |
rbac-job-launcher.yaml |
RBAC for provisioner K8s Jobs |
serviceaccounts.yaml |
K8s service accounts (portal, workers, provisioner) |
portal-backendconfig.yaml |
Portal health check + Cloud Armor attachment |
guacamole-backendconfig.yaml |
Guacamole Cloud Armor attachment |
ingress.yaml |
Public ingress, managed certificate, HTTPS redirect |
Values files¶
platform/charts/shifter/values.yaml- chart defaultsplatform/charts/shifter/values-gcp-dev.yaml-gcp-devoverridesplatform/charts/shifter/values-gcp-prod.yaml-gcp-prodoverrides- values that
gcp-deploygenerates - live Terraform outputs, runtime env, exact image identities, and edge policy names. Secret values are not in the Helm values.
Networking¶
Dual-network design, same pattern as AWS (see Networking).
| Network | Purpose |
|---|---|
platform |
GKE cluster, Cloud SQL (Private Services Access), Memorystore |
range |
Guest subnets for range instances. Cloud Router + NAT for egress. |
Networks are peered bidirectionally for platform-to-range connectivity.
Live-fire ranges use GCE range cells (per-range subnets with tag firewall isolation and explicit NAT enrollment) as the approved containment boundary per ADR-030 and ADR-039, with the containment model in ADR-056. Shared-VPC ranges use a provisioner-owned regional Router/NAT with only admitted subnets listed; VPC-per-range cells retain a range-owned Router/NAT. See the range-cell deploy runbook for capacity and lifecycle bounds. GDC custom L2 networks (VXLAN-based) for per-range guest isolation are a dev and operator-validation path only, not a live-fire containment boundary.
Deployment Path¶
One command stands up and updates a GCP deployment:
scripts/bootstrap/deploy.py gcp-deploy (ADR-070). The first install and each
update use the same command and the same Helm renderer. There are two ways to
run it:
- Without GitHub: an operator runs the deployer image of a published release. The command reads the deployment input bundle from the deployment config bucket and impersonates the purpose identities. See the GitHub-less GCP deployment guide.
- Through GitHub CI: a manual
deploy.ymldispatch calls_gcp-dev.yml. Each job renders the deployment input bundle from the GitHub secrets and variables, then runs the same command as its federated purpose identity.
The command runs these stages:
- load the deployment input bundle and take the deployment lock
- apply GCP Terraform, rewrite the live bucket objects with the bucket keys, and seed the first operator
- copy the release images into the deployment Artifact Registry (CI builds its images there)
- scan each exact image digest for HIGH and CRITICAL findings as the release-scan identity
- verify the provenance of each digest, render the Helm values, and run
helm upgrade --install --atomic - record the running images, then wait for the ingress address and the managed certificate
- run the advisory post-deploy smoke
gdc-bootstrap builds only the legacy GDC VM Runtime substrate for the gdc
range backend. It does not deploy the control plane.
Email delivery¶
GCP has no native Amazon SES equivalent, so a GCP deployment sends transactional mail (sign-up verification, password reset, invitations) through an operator-chosen SaaS, SendGrid or Mailgun, using django-anymail. The AWS deployment is unaffected: it keeps using django-ses over its IAM role.
Email is optional. With email_backend unset (the default), the portal uses Django's console backend and no email secret is created. To enable it:
- Set the email variables in the Terraform values of the deployment (
terraform.tfvarsorterraform.tfvars.jsonin the deployment input bundle):
# SendGrid
email_backend = "anymail.backends.sendgrid.EmailBackend"
email_from_address = "noreply@your-domain.example"
# or Mailgun (also needs the sender domain)
email_backend = "anymail.backends.mailgun.EmailBackend"
email_from_address = "noreply@your-domain.example"
email_sender_domain = "mg.your-domain.example"
-
Apply Terraform. This creates an unseeded Secret Manager secret
${project_prefix}-emailfor the ESP API key. The key is never stored in Terraform state, tfvars, Helm values, or this repo. -
Populate the secret with the API key as a JSON bundle, reading the key from a silent prompt so it never reaches shell history:
read -rs -p "ESP API key: " ESP_API_KEY && echo
printf '{"api_key":"%s"}' "$ESP_API_KEY" \
| gcloud secrets versions add "${PROJECT_PREFIX}-email" --data-file=- --project "$PROJECT_ID"
unset ESP_API_KEY
- Redeploy. The runtime renderer emits
EMAIL_BACKEND,DEFAULT_FROM_EMAIL, the secret referenceEMAIL_API_KEY_SECRET_ID, and (for Mailgun)MAILGUN_SENDER_DOMAIN;entrypoint.shhydrates the key intoEMAIL_API_KEYat startup, andconfig/_email.pyselects the backend.
If email_backend is set without email_from_address (or without email_sender_domain for Mailgun), the renderer fails closed at deploy time rather than sending as webmaster@localhost. See platform/terraform/gcp/README.md for the operator runbook.
Current gcp-dev concrete values¶
- Project:
example-gcp-project - Hostname:
shifter.example.com - Managed TLS: enabled
- Authorized admin CIDRs: empty; operator and CI access use Connect Gateway
Public operator egress IPs must not be added to
gke_master_authorized_cidrs; the private endpoint rejects them.
Related Docs¶
- Cloud Adapters - Protocol-based cloud abstraction
- Networking - VPC/network architecture (AWS section)
- Secrets - Secret management across clouds
- CI/CD - Full CI/CD documentation including GCP workflows
- Terraform - Terraform conventions and GCP module patterns