Secrets Management¶
Where secrets live and how to manage them.
Principles¶
- Never in code - No secrets in source files or environment configs
- GitHub Secrets for CI/CD - OIDC role ARNs plus the per-environment deploy payloads (
TF_VARS_*andSHIFTER_CONFIG_*) rendered at deploy time; see deploy-secrets.md. A GCP deployment that runs without GitHub keeps no deploy input in GitHub. Its deployment config bucket holds the inputs, and Secret Manager holds the secret values (ADR-070) - Cloud secret manager for runtime - AWS Secrets Manager today, GCP Secret Manager for
gcp-dev - Explicit, not default - No silent fallbacks; fail if secret missing
- Committed tfvars are non-sensitive baselines - The repo
terraform.tfvarsare fail-loudexample.complaceholders; real per-environment values live in theTF_VARS_*GitHub secrets and render into a gitignoredlocal.auto.tfvarsat deploy time (since #1250)
GitHub Secrets¶
These must be configured in repository Settings > Secrets and variables > Actions.
| Secret | Purpose |
|---|---|
AWS_ROLE_ARN |
GitHub Actions IAM role for prod (OIDC) |
AWS_ROLE_ARN_DEV |
GitHub Actions IAM role for dev (OIDC) |
GCP_DEPLOY_SERVICE_ACCOUNT |
GitHub Actions deploy service account email in the deployment Environment of a CI-deployed GCP tenant (none exists after #2668) |
GCP_PACKER_BUILD_SERVICE_ACCOUNT |
Image build service account in gcp-image-build; the foundation bootstrap writes it |
GCP_PACKER_VALIDATE_SERVICE_ACCOUNT |
Image validation service account in gcp-image-validate; the foundation bootstrap writes it |
GCP_RELEASE_SCAN_SERVICE_ACCOUNT |
Exact-release scanner service account in gcp-release-scan-<deployment suffix> |
GCP_WORKLOAD_IDENTITY_PROVIDER |
GitHub Actions workload identity provider resource of the GCP project that the Environment uses |
GCP_BOOTSTRAP_ADMIN_EMAIL |
Optional first GCP operator email for Identity Platform bootstrap |
GCP_BOOTSTRAP_ADMIN_PASSWORD |
Optional first GCP operator password for Identity Platform bootstrap |
PLATFORM_BOOTSTRAP_STAFF_EMAILS |
Optional comma-separated runtime staff bootstrap emails for production auth providers |
PLATFORM_BOOTSTRAP_SUPERUSER_EMAILS |
Optional comma-separated runtime superuser bootstrap emails for production auth providers |
TF_INFRA_STATE_BUCKET[_DEV] |
Terraform state bucket the backend render reads (prod is unsuffixed) |
TF_VARS_<ENV>_CORE / _RANGE / _PORTAL |
Per-environment Terraform local.auto.tfvars payloads, rendered to a gitignored local.auto.tfvars at deploy time (since #1250) |
SHIFTER_CONFIG_<ENV>_RANGE |
Deployment bigrae.yaml payload that renders the range egress allowlist |
SHIFTER_CONFIG_GCP_DEV |
Deployment bigrae.yaml of a CI-deployed GCP environment; the CI adapter renders it into the deployment input bundle |
AWS uses IAM role ARNs for OIDC federation. A CI-deployed GCP environment uses Google workload identity federation. A GCP deployment that runs without GitHub uses service-account impersonation by named principals instead. No static access keys are required.
Deploy tfvars (since #1250): the committed terraform.tfvars are fail-loud
example.com baselines. Each environment's real values live in the
TF_VARS_<ENV>_* secrets and are rendered into a gitignored local.auto.tfvars
at deploy time (*.auto.tfvars auto-loads over terraform.tfvars). Push the
AWS overlays with scripts/sync-deploy-secrets.sh. GCP inputs are not synced
there: publish them to the deployment config bucket with
scripts/bootstrap/deploy.py gcp-config publish. The single authoritative
per-environment checklist is
deploy-secrets.md.
AWS Secrets Manager¶
Runtime secrets accessed by the portal at startup.
Portal Secrets¶
| Secret Name | Contains |
|---|---|
shifter-{env}-portal-db-credentials |
RDS connection (host, port, user, password, dbname) |
shifter-{env}-portal-app |
Django SECRET_KEY, other app secrets |
shifter-{env}-portal-cognito |
OIDC client ID, client secret, domain |
shifter-{env}-portal-dc-domain |
Domain-controller domain Administrator password (DC_DOMAIN_PASSWORD), used by the engine provisioner to promote prebaked DC AMIs and to domain-join victims |
The domain-controller password is a live credential. It must not appear in committed tfvars, workflow YAML, Terraform plan comments, or command logs, only in Secrets Manager and (like every other portal secret) in restricted Terraform state.
Provisioning (Terraform-managed)¶
The Portal Terraform root creates and seeds the DC domain password secret.
Its random_password.dc_domain_password value is stored in Secrets Manager.
The AWS EKS provisioner environment carries the secret reference through the existing hydration boundary.
The secret is not stored in Helm values or committed configuration.
A fresh environment uses the normal infrastructure apply; no manual secret creation is required.
Dev Box¶
| Secret Name | Contains |
|---|---|
shifter-dev-box-admin-password |
Windows Administrator password (auto-generated) |
GCP Secret Manager¶
gcp-dev stages parallel runtime secret bundles with the same shapes the portal
already expects at startup:
| Secret Name | Contains |
|---|---|
shifter-gcp-dev-app |
Django SECRET_KEY, field encryption key |
shifter-gcp-dev-db |
Database connection JSON bundle |
shifter-gcp-dev-guacamole-db |
Guacamole PostgreSQL connection JSON bundle |
shifter-gcp-dev-guacamole-json-auth |
Guacamole JSON auth signing key |
Current rollout behavior:
shifter-gcp-dev-appandshifter-gcp-dev-dbare seeded by Terraform for the first deployable control-plane sliceshifter-gcp-dev-guacamole-dbandshifter-gcp-dev-guacamole-json-authare seeded by Terraform and synced into theguacamole-runtimeKubernetes Secret out of band during deploy (kubectl applyfrom Secret Manager, before the Helm release). Secret Manager and the Kubernetes Secret are the only runtime copies: these values are never placed in Helm chart values or Helm release history. The chart references theguacamole-runtimeSecret by name only (envFrom.secretRef).- Identity Platform is provisioned by Terraform for the secure GCP portal login path
gcp-deployseeds the first GCP operator fromsettings.operator_bootstrap: theadmin_email, and the password in the Secret Manager secret thatadmin_password_secret_refnames. CI suppliesGCP_BOOTSTRAP_ADMIN_EMAILthrough the CI adapter and passesGCP_BOOTSTRAP_ADMIN_PASSWORDthrough--operator-password-env. The password never enters Terraform, the deployment input bundle, or a command argument- Bootstrap operator elevation comes from
settings.operator_bootstrap.staff_emails/superuser_emails, which the deploy renders intoPLATFORM_BOOTSTRAP_STAFF_EMAILS/PLATFORM_BOOTSTRAP_SUPERUSER_EMAILS. CI maps those GitHub secrets onto the settings. These values must stay out of committed source - The same
PLATFORM_BOOTSTRAP_STAFF_EMAILS/PLATFORM_BOOTSTRAP_SUPERUSER_EMAILScontract is applied by the AWS/Cognito OIDC backend and the GCP Identity Platform backend on user login. These settings identify which authenticated users receive Djangois_staff/is_superuser; they are not Django passwords and do not bypass the configured identity provider. - GCP corporate users authenticate in the browser through Identity Platform's FirebaseUI/SDK flow. Django only accepts verified Google identity tokens and creates the application session after email-verification and MFA checks pass.
- The GCP deploy command assumes the secure portal posture. It sets the public hostname from
deployment.domainand always enables managed TLS - The operational dependency chain is now explicit:
public_hostnamemust be configured- managed TLS must be enabled
- Identity Platform must be provisioned successfully
- the public DNS record must point at the GKE ingress IP so the managed certificate can become active
What's NOT a Secret¶
These are configuration, not credentials:
- AWS region
- Instance types
- AMI IDs
- S3 bucket names
- VPC CIDRs
- Domain names
- Feature flags
They do not grant access or authenticate. Since #1250, though, their real
per-environment values are not committed either: the repo terraform.tfvars are
example.com baselines, and the live values (including domains, bucket names,
and CIDRs) ship in the TF_VARS_* secrets rendered to a gitignored
local.auto.tfvars at deploy time (see deploy-secrets.md).
Rule: If it doesn't grant access or authenticate, it's config, not a secret.
Adding a New Secret¶
GitHub Secret¶
- Go to Settings > Secrets and variables > Actions
- Click "New repository secret"
- Name it following existing conventions
- Reference in workflow:
${{ secrets.SECRET_NAME }}
AWS Secrets Manager¶
-
Create via Terraform (preferred):
-
Or via CLI:
-
Grant access via IAM policy in Terraform
- Access in code via boto3 or environment variable injection
Rotating Secrets¶
The bullets below are high-level operator notes, not a full production rotation
strategy. The production rotation policy per credential class is
docs/architecture/secrets-rotation-strategy.md; before changing production
rotation behavior, apply the architecture guardrails in
docs/architecture/secrets-rotation-strategy-preflight-159.md.
Database Password¶
The current portal reads DB_PASSWORD at container startup. Rotating the
database password must keep the actual RDS password and the Secret Manager JSON
bundle consistent, then restart/drain portal processes so they hydrate the new
value. Updating only one side is not a completed rotation.
Django SECRET_KEY¶
Updating django_secret_key in the app secret and restarting the portal
invalidates existing sessions unless explicit SECRET_KEY_FALLBACKS support is
added first. Do not rotate field_encryption_key as a side effect of rotating
django_secret_key.
Cognito Client Secret¶
Treat this as a coordinated Cognito app-client credential swap. Keep old and new client credentials available through portal deploy/drain; do not delete the old client while old portal processes can still use it.
VM guest passwords (per-instance)¶
Range guest desktop / RDP passwords are unique per instance. They are not stored in the platform secret bundles, not carried as environment variables, and not baked into AMIs. The password value never appears in EC2 user_data, IMDS, Terraform-rendered cloud-init manifests, or the guest's process argv.
The lifecycle aligns with the documented best practice for AWS Systems
Manager Parameter Store + Run Command (and the GCP equivalent: control-
plane-pushes-after-boot via the per-instance SSH identity-brokered
channel). See docs/architecture/vm-guest-credential-preflight-762.md.
Lifecycle¶
- Generate: the range Terraform module (AWS) and
gdc_vmruntime_assets._ensure_rdp_password_secret(GCP) generate a per-instance random password with all four character classes (upper, lower, digit, shell-safe punctuation) enforced. - Store: the value is persisted in the cloud's secret manager:
- AWS:
arn:aws:secretsmanager:<region>:<account>:secret:shifter/<env>/range/<range_id>/<role>-<uuid8>-rdp-password - GCP:
projects/<project>/secrets/shifter-<env>-range-<range_id>-<role>-<token>-rdp-password - Reference, don't render: the secret ARN / resource path flows
into
engine_instance.state.rdp_password_secret_arnandRange.provisioned_instances[].rdp_password_secret_arn. The value never lands in EC2 user_data, GDC VM Runtime cloud-init data, or Terraform-rendered scripts. - Push at provisioning time: after the instance comes up and the
per-instance SSH key is in
authorized_keys/administrators_authorized_keys, the engine provisioner runsSetLocalPasswordPlanviaSetupOrchestrator: - Linux (kali, ubuntu): the password is piped through
chpasswdstdin, never as achpasswdargv. - Windows victim:
Set-LocalUser -Password (ConvertTo-SecureString …), never as anet userargv. SetupOrchestrator.SENSITIVE_CONTEXT_KEY_PARTSmasks the value in captured stdout/stderr because the context key isrdp_password.- Resolve at access time: the portal's
engine.services._resolve_non_dc_rdp_passwordreads the per- instance secret reference fromRange.provisioned_instancesand fetches the value viaengine.secrets.get_rdp_password→shared.cloud.get_secrets_store().get_secret(). If the fetch fails (deleted secret, IAM regression), the portal raises a non-sensitiveValueErrorthat the Mission Control RDP view maps to HTTP 400, the same envelope as a missing reference.
RAES-authored account credentials¶
The RAES-native GCE path treats authored accounts separately from the
provisioner's raes management login and from portal/participant credentials.
For every enabled account on every concrete node instance, the provisioner
read-or-creates a deterministic Secret Manager entry keyed by range, instance,
username, and canonical auth_method:
passwordgenerates according to the authoredweak,medium, orstrongpolicy and reusesSetLocalPasswordPlanto install and verify it.publickeystores the generated private key and sends only the public half to an account-specificauthorized_keyspath. Windows receives aMatch Useroverride so an administrator account never falls through to the sharedadministrators_authorized_keysfile.- Disabled accounts receive no usable credential.
password_strength: noneis accepted only for a disabled account, never as a blank-password login.
Credential values and references do not enter the serialized RAES plan, GCE
metadata, provisioner outputs, runtime snapshots, events, diagnostics, or logs.
The existing strict-host-key-checked management SSH channel carries setup after
boot, and SetupOrchestrator masks password context. Destroy reconstructs and
deletes each per-account secret. There is intentionally no portal or API
retrieval surface for these backend-owned credentials.
auth_method admits only the canonical password and publickey spellings at
both the shared admission boundary and the separate provisioner parser. The
backend does not declare RAES mail: Linux aliases and Windows marker files do
not provide one equivalent, verified mail-routing semantic.
DC role¶
The DC role keeps the deployment-scoped DC_DOMAIN_PASSWORD contract
documented below: a DC host's local Administrator account is the
domain Administrator account, so per-instance rotation is handled
through DC promotion rather than the per-instance secret.
Residual state exposure¶
The generated random_password.result (AWS) and the
add_secret_version payload (GCP) appear in Terraform / provisioner-
side memory at apply time and in the Terraform state object for the
AWS path, the same as the existing per-instance SSH-key precedent
(tls_private_key.instance and aws_secretsmanager_secret_version.ssh_key).
Terraform state lives in the S3 backend with bucket-level encryption
and least-privilege IAM access; this is the established mitigation.
Eliminating state exposure entirely would require external secret
generation (for example, a Lambda invoking aws secretsmanager create-secret)
and is a separate workstream.
AWS delivery channel¶
After bootstrap, the provisioner reads the guest's ed25519 SSH host public key over the authenticated SSM control channel. It then pins that key, retrieves the per-instance password and management private key from Secrets Manager, and sends the password only over the pinned SSH connection's stdin. The non-secret setup script travels separately. The password therefore does not enter EC2 user data, SSM Run Command history, script source, process argv, environment, or metadata. The guest instance role has no permission to read the secret store.
Do not replace this with {{ssm-secure:...}}: that syntax is a CloudFormation
dynamic reference and Systems Manager Run Command document parameters do not
support SecureString Parameter Store references.
Rotation¶
Per-instance rotation is achieved by destroying and re-provisioning
the range; the Terraform random_password regenerates, the new value
replaces the previous secret version, and SetLocalPasswordPlan pushes
it on the next provision. Image rebuilds (Packer) do not rotate
live credentials; they only refresh the bootstrap scripts shipped on
first boot.
Domain Controller Administrator Password¶
The portal Django container reads DC_DOMAIN_PASSWORD once at startup
(entrypoint.sh) and caches it in the process environment, so a Secrets
Manager update does not refresh a running portal. The engine provisioner ECS
task, by contrast, has the value injected by ECS at each task launch via the
task definition's secrets = [...] block, so range provisions/joins after the
rotation pick up the new value without a task-definition redeploy.
- Regenerate the value and re-seed the secret with one Terraform apply:
(Terraform regenerates
random_password.dc_domain_passwordand updatesaws_secretsmanager_secret_version.dc_domain_password.) - Restart the portal containers so the new value is loaded: (or terminate the portal EC2 and let the ASG relaunch).
- Re-provision any ranges whose DC was promoted with the old value so the live domain credential matches the secret again; existing ranges keep the old password until they are re-provisioned. No engine provisioner task-definition redeploy is required; the next ECS task launch reads the rotated value.
GitHub OIDC Role¶
Role ARN doesn't change. Trust policy updates are in platform/terraform/global/iam/.
GCP Workload Identity Federation¶
The provider resource and target service account stay stable. Rotate or tighten permissions in GCP IAM without changing the GitHub workflow contract unless the provider or service account identity itself changes.
Federation is optional (ADR-070). A GCP deployment that runs without GitHub
grants roles/iam.serviceAccountTokenCreator on each purpose identity only to
the principals in the impersonation_principals input of the cicd-oidc root.
Change that list to add or remove an operator. No service-account key exists.
Accessing Secrets Locally¶
For local development, you can retrieve secrets:
# Get a secret value
aws secretsmanager get-secret-value \
--secret-id shifter-dev-portal-db-credentials \
--profile $PANW_SHIFTER_DEV_PROFILE \
--query SecretString --output text | jq .
# Use in environment
export DB_PASSWORD=$(aws secretsmanager get-secret-value \
--secret-id shifter-dev-portal-db-credentials \
--profile $PANW_SHIFTER_DEV_PROFILE \
--query SecretString --output text | jq -r .password)
Security Checklist¶
- [ ] Secret not in any committed file
- [ ] Secret not in shell history (
use read -sfor input) - [ ] Appropriate IAM policy restricts access
- [ ] Rotation procedure documented
- [ ] Dev and prod use different secrets