Skip to content

GCP model broker onboarding and qualification probes

Owner: deployment operator with Engine/security owners. Scope: M06 / #2123, PLAT-202. These instructions produce evidence; their presence is not evidence that a deployment passed. M10 / #2127 owns the independent runtime result. Keep the broker disabled until the M05 executable, M07 adapter and M08 admission/enrollment path are available in the attested release image.

Project and identity onboarding

A model project may be the platform project, the dynamic-secret project, or a dedicated project (#2243 source-management boundary). Broker and invocation identities stay distinct when projects coincide.

Each model project needs billing enabled and the shifter-deployment label equal to deployment.name. Root settings.model_broker.model_projects is the reviewed project-to-invocation account inventory. Confirm regional model availability, quota and organization-policy compatibility with the approved catalog before enabling a cohort. The deployment operator needs authority to enable APIs and create the exact GSAs, custom roles and bindings; the broker does not inherit that operator authority.

After a reviewed Terraform apply, save output in a restricted local directory. Terraform outputs can include sensitive values: do not print, upload or commit the complete output. The compatibility renderer compares applied broker coordinates with root intent before applying workload manifests. Run the read-only project and effective-IAM probe using the broker GSA from that readback:

uv run --project scripts/gcp python scripts/gcp/probe_model_broker.py \
  --config bigrae.yaml \
  --broker-gsa '<applied-broker-gsa>' \
  --output model-project-onboarding.json

The probe checks ownership, billing, APIs, live invocation identity, absence of user-managed keys, exact-target token permission and invocation permission. It requires conclusive denials for management/key/actAs and representative compute, secret, storage and endpoint-administration permissions. Unknown or conditional Policy Troubleshooter results fail qualification. The operator needs permission to inspect effective policy, including inherited policy. Project-level negatives do not prove absence of resource-specific grants: M10 must also inventory applicable resource policies and test actual denied resources. A successful probe never proves regional model availability or absence of every possible permission. Retain only its bounded result and reviewed policy inventory, not credentials or raw provider responses.

Private transport and source preservation

Use the actual deployment project and region, never infer them from an account name. Resolve the project from the deployment configuration (the GCP_PROJECT_ID for the target environment) and export it, for example export GCP_PROJECT=<your-project>; the region is us-central1. Read-only cluster inspection:

gcloud container clusters describe shifter-gcp-dev-gke \
  --project "$GCP_PROJECT" --region us-central1 \
  --format='yaml(currentMasterVersion,network,subnetwork,networkConfig.datapathProvider,privateClusterConfig.enablePrivateNodes,workloadIdentityConfig.workloadPool)'
kubectl -n shifter-platform get service model-broker -o yaml
kubectl -n shifter-platform get networkpolicy -o yaml
kubectl -n shifter-platform get endpointslice \
  -l kubernetes.io/service-name=model-broker -o yaml

Verify the kubeconfig points at that cluster before probing. Record release SHA and image digest, GKE/CNI version, internal forwarding rule, LB backend health and realized firewall targets/source ranges. The forwarding rule must use the reserved VIP on TCP 443, internal passthrough semantics and the configured regional/global-access setting. The Service must use externalTrafficPolicy: Local; inspect both nodes hosting replicas. Evaluate all selecting NetworkPolicies and VPC/hierarchical rules together. Controller health checks may reach their health-check port; they must not become a participant management or node-port grant.

On each selected VM, inspect the instance with gcloud compute instances describe and a field-limited format for canIpForward, service accounts, network interfaces and tags. Require canIpForward=false, no service account, no external IP, a current Engine-owned subnet binding and only the expected range tags. Machine-image inheritance and alternate NICs require actual readback. Forwarding/VPN hosts and NAT/proxy paths are unqualified.

Run these cases from two independently admitted ranges and one unadmitted subnet. Use the M05/M08 supported client and a bounded synthetic profile; never put bearer tokens in shell arguments, history or evidence. Store client credentials in an operator-approved private file or stdin mechanism. Use verified TLS with the configured hostname and deployment CA, including when pinning the VIP for a probe. Never use insecure verification flags.

Probe Required observation
Range A calls broker VIP:443 with its current capability Verified TLS; Engine admits A; broker socket peer equals A's actual private source and belongs to Engine's current subnet binding.
Repeat against both replicas and across a draining rollout The same source identity survives the actual LB/CNI path; no node/NAT address replaces it.
Range B replays A's capability Rejected before provider dispatch or accounting charge, even with forged Forwarded / X-Forwarded-For naming A.
Unadmitted subnet calls VIP:443 Network denial; no broker admission.
A connects to VIP:80, :8443 or :8444, neighboring VIP, peer range, node/pod management address Denied; only exact broker VIP:443 is the new capability.
Strict none range requests broker enrollment Admission/projection rejected; no NAT or broker egress exception.
Broker attempts database, Redis, Kubernetes API or arbitrary internet destination Denied by effective policy; narrow Engine TLS/DNS/identity/provider paths remain available.
TLS hostname mismatch, untrusted/expired certificate or wrong control audience/subject Rejected with no insecure fallback or paid provider probe.

Source comparison unit tests cannot replace these observations. Export only case IDs, safe binding references, pass/fail, timings and release identity. Do not record tokens, prompts, completions, headers, body hashes or raw error responses. If the live path does not preserve the peer, leave admission off and repair the topology; forwarding headers are never an alternative identity.

Rotation and release

Create separate versioned broker/control TLS Secrets and a versioned CA ConfigMap through the existing secret-management procedure. Broker SAN is the configured private hostname; control SAN includes model-access-control.shifter-platform.svc. Never route certificate material through Terraform or ordinary chart values. Renew before expiry and alert on renewal failures. Change root Secret references to roll replicas; do not assume that updating mounted bytes reloads the server TLS context.

For CA replacement, distribute overlapping trust first, roll both servers and guests, verify new certificates and active streams, then remove old trust only after old connections and pods drain. Probes must remain independent of paid providers. Measure bounded drain and revocation with a synthetic stream; 150 seconds of pod grace includes endpoint withdrawal and the design's 120-second request maximum. Missing renewal, source evidence or authorization service means admission stays disabled. Rollback drains/disables model access and never restores guest provider credentials.

Attach project/IAM onboarding, safe transport case results, TLS rotation, revocation/stream evidence and exact release identity to #2127. Operator approval of those results precedes cohort enablement. See the deployment package and operating design for ownership and migration boundaries.