Skip to content

GCP Infrastructure

GKE-based deployment of the BigRAE platform on Google Cloud.

Directory Structure

platform/
├── terraform/gcp/
│   ├── modules/platform-core/       # All GCP platform infrastructure
│   ├── environments/platform/       # Shared platform root for every deployment
│   ├── environments/<deployment>/   # Committed values of the CI-deployed tenants
│   └── global/                      # cicd-oidc and github-runner roots
├── charts/shifter/                  # Helm chart: the only GCP Kubernetes renderer
└── deployer/gcp/                    # Deployer image of one published release

Terraform provisions the GCP resources. The Helm chart packages the BigRAE control plane. The deploy command (scripts/bootstrap/deploy.py gcp-deploy) renders the generated values from the Terraform outputs, the root config, and the Secret Manager payloads. Then it installs the chart. A deployment that runs without GitHub keeps its own Terraform values in its deployment input bundle, not in this repository.

Terraform Module: platform-core

Single module provisions the entire GCP control plane.

Resource Service Purpose
VPC Networks (×2) VPC platform (GKE, shared services) and range (guest isolation)
GKE Cluster GKE Private nodes, VPC-native, Workload Identity enabled, Binary Authorization enforced via project singleton policy, public control-plane endpoint restricted by authorized CIDRs
Node Pools (×3) GKE web (portal, Guacamole), workers (domain workers), provisioner (range provisioning jobs)
Cloud SQL Cloud SQL PostgreSQL. Hosts platform DB and Guacamole DB. Private IP only.
Memorystore Memorystore Redis. Channel layer and worker coordination.
Pub/Sub Pub/Sub Event topic with per-domain subscriptions (cms, engine, mc).
Artifact Registry Artifact Registry Container repositories (portal, guacd, guacamole-client, pulumi-provisioner).
Secret Manager Secret Manager Runtime secret bundles (app, db, guacamole-db, oidc, guacamole-json-auth).
Cloud DNS Cloud DNS Optional. Public hostname with Google-managed TLS certificate.
Cloud Armor Cloud Armor Baseline WAF policy attached to the public portal and Guacamole GKE backends.
GCS Buckets Cloud Storage Assets/artifacts and Terraform state. Each Terraform-owned bucket uses its own customer-managed Cloud KMS key (Bucket encryption keys).
Service Accounts IAM Separate accounts for GKE nodes and workloads (portal, workers, provisioner). Workload Identity binds K8s SAs to GCP SAs.

GKE Cluster

Private nodes with no public node IPs. Cloud NAT provides outbound connectivity.

  • Release channel: REGULAR (automatic patching)
  • Workload Identity: Enabled. Pod service accounts map to GCP service accounts.
  • Networking: VPC-native with secondary IP ranges for pods and services
  • Logging/Monitoring: System components and workload logging enabled
  • Control-plane access: private endpoint reached through Connect Gateway; optional direct access accepts connected RFC1918 networks only
  • Binary Authorization: enabled with PROJECT_SINGLETON_POLICY_ENFORCE

Node Pools

Pool Role Notes
web Portal and Guacamole Handles user-facing traffic
workers CMS, Engine, MC background workers Queue consumers
provisioner Range provisioning jobs Runs K8s Jobs for range lifecycle

Machine types and node counts are configurable per environment.

Kubernetes Workloads

Managed via Helm with the chart defaults, the profile values, and the values that gcp-deploy generates.

Chart Resources (platform/charts/shifter/templates/)

Manifest Workload
web-deployment.yaml Portal Django app
guacd-deployment.yaml Guacamole protocol daemon
guacamole-client-deployment.yaml Guacamole web client
worker-cms-deployment.yaml CMS queue consumer
worker-engine-deployment.yaml Engine queue consumer
worker-mc-deployment.yaml Mission Control queue consumer
ctf-scheduler-deployment.yaml CTF batch scheduler
rbac-job-launcher.yaml RBAC for provisioner K8s Jobs
serviceaccounts.yaml K8s service accounts (portal, workers, provisioner)
portal-backendconfig.yaml Portal health check + Cloud Armor attachment
guacamole-backendconfig.yaml Guacamole Cloud Armor attachment
ingress.yaml Public ingress, managed certificate, HTTPS redirect

Values files

  • platform/charts/shifter/values.yaml - chart defaults
  • platform/charts/shifter/values-gcp-dev.yaml - gcp-dev overrides
  • platform/charts/shifter/values-gcp-prod.yaml - gcp-prod overrides
  • values that gcp-deploy generates - live Terraform outputs, runtime env, exact image identities, and edge policy names. Secret values are not in the Helm values.

Networking

Dual-network design, same pattern as AWS (see Networking).

Network Purpose
platform GKE cluster, Cloud SQL (Private Services Access), Memorystore
range Guest subnets for range instances. Cloud Router + NAT for egress.

Networks are peered bidirectionally for platform-to-range connectivity.

Live-fire ranges use GCE range cells (per-range subnets with tag firewall isolation and explicit NAT enrollment) as the approved containment boundary per ADR-030 and ADR-039, with the containment model in ADR-056. Shared-VPC ranges use a provisioner-owned regional Router/NAT with only admitted subnets listed; VPC-per-range cells retain a range-owned Router/NAT. See the range-cell deploy runbook for capacity and lifecycle bounds. GDC custom L2 networks (VXLAN-based) for per-range guest isolation are a dev and operator-validation path only, not a live-fire containment boundary.

Deployment Path

One command stands up and updates a GCP deployment: scripts/bootstrap/deploy.py gcp-deploy (ADR-070). The first install and each update use the same command and the same Helm renderer. There are two ways to run it:

  • Without GitHub: an operator runs the deployer image of a published release. The command reads the deployment input bundle from the deployment config bucket and impersonates the purpose identities. See the GitHub-less GCP deployment guide.
  • Through GitHub CI: a manual deploy.yml dispatch calls _gcp-dev.yml. Each job renders the deployment input bundle from the GitHub secrets and variables, then runs the same command as its federated purpose identity.

The command runs these stages:

  1. load the deployment input bundle and take the deployment lock
  2. apply GCP Terraform, rewrite the live bucket objects with the bucket keys, and seed the first operator
  3. copy the release images into the deployment Artifact Registry (CI builds its images there)
  4. scan each exact image digest for HIGH and CRITICAL findings as the release-scan identity
  5. verify the provenance of each digest, render the Helm values, and run helm upgrade --install --atomic
  6. record the running images, then wait for the ingress address and the managed certificate
  7. run the advisory post-deploy smoke

gdc-bootstrap builds only the legacy GDC VM Runtime substrate for the gdc range backend. It does not deploy the control plane.

Email delivery

GCP has no native Amazon SES equivalent, so a GCP deployment sends transactional mail (sign-up verification, password reset, invitations) through an operator-chosen SaaS, SendGrid or Mailgun, using django-anymail. The AWS deployment is unaffected: it keeps using django-ses over its IAM role.

Email is optional. With email_backend unset (the default), the portal uses Django's console backend and no email secret is created. To enable it:

  1. Set the email variables in the Terraform values of the deployment (terraform.tfvars or terraform.tfvars.json in the deployment input bundle):
# SendGrid
email_backend      = "anymail.backends.sendgrid.EmailBackend"
email_from_address = "noreply@your-domain.example"

# or Mailgun (also needs the sender domain)
email_backend       = "anymail.backends.mailgun.EmailBackend"
email_from_address  = "noreply@your-domain.example"
email_sender_domain = "mg.your-domain.example"
  1. Apply Terraform. This creates an unseeded Secret Manager secret ${project_prefix}-email for the ESP API key. The key is never stored in Terraform state, tfvars, Helm values, or this repo.

  2. Populate the secret with the API key as a JSON bundle, reading the key from a silent prompt so it never reaches shell history:

read -rs -p "ESP API key: " ESP_API_KEY && echo
printf '{"api_key":"%s"}' "$ESP_API_KEY" \
  | gcloud secrets versions add "${PROJECT_PREFIX}-email" --data-file=- --project "$PROJECT_ID"
unset ESP_API_KEY
  1. Redeploy. The runtime renderer emits EMAIL_BACKEND, DEFAULT_FROM_EMAIL, the secret reference EMAIL_API_KEY_SECRET_ID, and (for Mailgun) MAILGUN_SENDER_DOMAIN; entrypoint.sh hydrates the key into EMAIL_API_KEY at startup, and config/_email.py selects the backend.

If email_backend is set without email_from_address (or without email_sender_domain for Mailgun), the renderer fails closed at deploy time rather than sending as webmaster@localhost. See platform/terraform/gcp/README.md for the operator runbook.

Current gcp-dev concrete values

  • Project: example-gcp-project
  • Hostname: shifter.example.com
  • Managed TLS: enabled
  • Authorized admin CIDRs: empty; operator and CI access use Connect Gateway

Public operator egress IPs must not be added to gke_master_authorized_cidrs; the private endpoint rejects them.

  • Cloud Adapters - Protocol-based cloud abstraction
  • Networking - VPC/network architecture (AWS section)
  • Secrets - Secret management across clouds
  • CI/CD - Full CI/CD documentation including GCP workflows
  • Terraform - Terraform conventions and GCP module patterns