Model-access operations design¶
Current support: ADR-068 parks the broker; keyless-direct is the supported guest model path. These broker procedures are historical implementation guidance and do not authorize activation. The #2274 replacement evaluation defines qualification for the selected LiteLLM OSS gateway, including deployment-owned key rotation, automatic private TLS renewal and fresh bootstrap; it is not an installed service.
ADR-066's identity cutover closes the retained sharing API and its composition
entry points before membership reads or publication effects. They remain
unavailable even when MODEL_ACCESS_ENABLED is set. This is enforcement of
ADR-068, not gateway activation. Do not use these historical sharing procedures
to enable the parked broker.
Status: proposed for #681, ADR-061, 2026-09-06. Procedures describe implementation requirements. Management endpoints and operational commands must be delivered and tested by the owning issues before use. No live provider, load, revoke, migration or restore evidence is asserted here.
Deployment and responsibility¶
Run a dedicated broker Deployment in the existing BigRAE Helm release, using a separate service account, private TLS service and bounded resource requests/limits. Start qualification with two replicas on separate nodes where available, a disruption budget, graceful draining and pinned images. Use the existing Cloud SQL/PostgreSQL, Engine worker, GKE monitoring/logging, shared audit and deployment pipeline. Do not introduce a Redis authority, gateway database, service mesh, per-range model project or new queue service.
| Owner | Operational responsibility |
|---|---|
| Deployment/cloud operator | Billing/model enablement, approved project/account inventory, IAM, egress, TLS, quota requests and emergency shard identity disable. |
| Platform operator | Catalog and price review, account ceilings, broker rollout, status, revoke/re-enroll, reconciliation, support and recovery. |
| Event organizer | Accurate cohort/spare demand and event window, delegated model profile, event budget, participant communications and authorized event access suspension. |
| Scenario maintainer | Pinned client/protocol/artifact compatibility, required/optional model behavior and useful participant acceptance. |
| Release/security owner | Independent IAM/network/adverse-case evidence, documented operating envelope, exceptions and release decision. |
Configuration flows from installation validation through Terraform outputs, runtime rendering, settings/env manifest and chart values. Secrets use exact references and workload identity; a setup script must not become a second catalog. Register the new production process in path ownership, test/security CI routing, SBOM/image provenance and import checks. Keep the exact provisioner-Job admission shape intact.
Readiness checks configuration/schema, supported adapter versions, control authentication and accounting reachability; it does not send a paid prompt. Liveness checks process health without depending on provider availability. An authenticated, rate-limited synthetic probe has its own small operator budget and fixed harmless prompt. It exercises the real data path before a cohort is enabled. Its result cannot make an unauthorized range grant active.
TLS uses a deployment-approved CA or publicly trusted certificate for the private hostname, automated renewal and overlap. Guests verify the hostname and chain from their qualified image. Never disable verification. Monitor certificate expiry and test rotation with active streams. The GCP private load balancer must preserve the source identity required by the security design; rendered manifests alone do not prove this.
Objectives and capacity envelope¶
These are initial release acceptance targets to measure, not current SLOs:
| Dimension | Proposed target and measurement |
|---|---|
| Broker/control availability | 99.9% over the declared event window for valid, within-budget requests, measured separately from upstream availability and policy denials. |
| Added admission latency | p95 at most 250 ms and p99 at most 1 s within one deployment region, excluding provider generation; record cohort size and database contention. |
| Revocation | New admission denied after the Engine commit; downstream delivery and upstream transport stopped within 10 s. Measure last byte and dispatch attempts, including control-service loss. |
| Request bounds | Default total 120 s, connect 5 s, stream idle 30 s, control timeout 2 s and continuation checks at most 5 s apart. A profile may tighten these; larger bounds require measured qualification and revised maximum charge. |
| Payload and process bounds | Initial 1 MiB decompressed JSON request, 32 KiB headers, 1 MiB maximum SSE event and 256 KiB streaming buffer per connection; reject excess. Pin depth/field-count/response limits in the adapter contract and test the selected client's useful context. |
| Reconciliation | Expired dispatch/settlement leases discovered within 60 s; unknown charges remain reserved or conservatively spent, never automatically refunded. |
| Recovery | Restore exercise target: accounting RPO at most 5 min and admission recovery within 60 min after database availability, with mandatory fencing and reconciliation. Older restored state cannot simply regain remaining budget. |
Sizing inputs are measured active ranges, concurrent requests per range,
request arrival rate, output rate, mean/p95 duration, token upper bounds,
provider pool limits, broker CPU/memory and Engine/DB request latency. For
N active ranges with at most C connections each, test at least N*C
broker transports and the corresponding reserve/settle plus continuation
traffic (up to N*C/5 continuation checks per second at full utilization).
Admission and pool row contention can be the first limit; adding broker
replicas does not increase provider quota or remove database serialization.
Size against effective sharing pools as well as range counts. Test one large all-ranges pool, many individual pools and overlapping group/event/user bindings. Shared capacity is committed once with child draws; dedicated reservations still add against real provider quota. A common budget row can become the contention limit. Exercise bounded selector pagination, dynamic membership bursts and synchronous revision fencing before background reassessment; do not qualify only fixed event-sized snapshots.
Use bounded tests at the declared cohort and a controlled overload case. Report the maximum sustainable cohort from evidence. The design makes no 200-desktop guarantee. HPA may respond to active streams and resource usage within operator limits; scale-down drains connections for the request bound and then terminates them with retained accounting. There is no transparent stream migration or post-crash replay.
Scenario model needs and required-access admission (M02, #2119)¶
A scenario declares its logical model need through a BigRAE-owned, staff-authored
overlay (ScenarioModelNeeds), not by editing the RAES pack and not through the
deploy-time mounted catalog. Author it per registered scenario pack:
- Key it by
scenario_idand setauthored_package_digestto the exact pack content digest the need is authored against. Map eachworkload_roleto aScenarioNeed(required flag, required/allowed capabilities, allowed strategies, data regions, limits) whoseprofile_idnames a profile in the mounted catalog. The overlay is validated on save; a workload-role/key or digest mismatch is rejected. - Because the overlay rides the runtime pack lifecycle, adding a model-requiring
pack needs no catalog redeploy. But re-registering the pack with new content
changes its digest: re-author the overlay against the new digest, or a
required launch fails closed (
digest_mismatch) until you do.
Organizers declare typed model demand per workload on the event (CTF-908): expected concurrency and per-participant request/input/output demand plus the allowed strategy. This is capacity/sizing input constrained by the scenario need; it cannot name a provider, account, region, shard, credential, or price.
Required-access failure is deliberate and visible. A launch of a scenario with a
required model need is refused before any range is dispatched, across every
launch family (participant, spare, wave, standalone, recovery-rebuild, and
warm-claim activation), when: model access is unconfigured, the referenced
profile is absent, the pack digest is stale, a required capability is
unavailable, the profile intersection is empty, the sharing overlap conflicts,
or the range posture is zero-egress (deny-all/none) for required external
model use. An unavailable sharing authority yields an indeterminate outcome that
also refuses the launch (fail closed) rather than proceeding. Optional model
access that cannot be satisfied is admitted as an explicit visible absence and
does not block. The refusal carries a bounded reason code and no raw figures or
rejected values. The admitted private-broker exception for a zero-egress range
is deferred with the broker runtime.
Cost and quota controls¶
Private broker runtime rollout (M05)¶
Apply Engine migration 0086_model_credential_rotation_budget before deploying
the updated control process. It adds a nullable window start and zero-initialized
rotation count; existing opaque tokens and original hard deadlines are retained.
For an active GCP deployment, refresh the applied Terraform output and render
broker_subject_id / provisioner_subject_id from the actual service-account
unique IDs. Older email-only active output is deliberately rejected. Disabled
and standby deployment behavior is unchanged. These non-secret IDs are not
provider keys and must not be supplied from guest or tenant request input.
Broker readiness now checks authenticated Engine/DB reachability. SIGTERM fences active streams before normal server shutdown; retain control and reconciliation capacity while draining. Expect fail-closed load shedding when worker/rotation budgets are exhausted, and preserve unknown holds after interrupted transport. Use authored error codes and accounting state for diagnosis, not SDK wire DEBUG logging, request dumps or captured prompts. The exact runtime bounds and local test posture are recorded in the broker contract. No live provider, source-preservation, load or restore qualification is claimed.
Before an event, inventory provider model enablement, supported regions, quota-pool dimensions, observed usage/freshness, expected input/output rates, shard weights, capacity reservations and reviewed price validity. Test one approved call per configured alias/project. Distinguish published limits, observed limits, deployment caps and unobservable/dynamic capacity. Quota values and model access are deployment prerequisites, not universal defaults; see Google's current quota guidance.
Event maximum authorized model cost is the sum of permitted request upper bounds, capped by all parent budget accounts. Actual provider invoices may include tariff changes, taxes, infrastructure and non-broker use. Budget alerts are useful independent detection, not hard admission. Enable provider billing alerts at operator-chosen thresholds and reconcile bounded aggregate usage against local accounting without exporting prompt bodies.
Budget for two broker replicas, internal load balancing, DNS/TLS, metrics/log retention, PostgreSQL I/O/storage, network/NAT egress, token issuance, optional provider committed throughput and inference/tool charges. Avoid quoting a fixed monthly price without deployment quantities and current tariffs. Cross-project sharding has IAM and quota administration cost; separate keys do not buy additional quota. Reuse the existing deployment's infrastructure until measured load requires a reviewed change.
Request accounting and reconciliation (M04)¶
Engine reserves a conservative upper charge against every applicable spend,
rate and concurrency account before a request can dispatch, and settles the
proven usage once against the request's immutable price snapshot. A timeout,
disconnect, missing usage or crash after possible provider work leaves the
request unknown and retains the conservative hold; the request never replays.
The request-accounting reconciliation pass rides the existing Engine reconciler
(no separate daemon): at each obligation's deadline it moves an unknown hold
from reserved to conservatively spent at the same value and closes expired
revocation fences. It never refunds a hold on a timer, invokes a provider, or
replays a request. Later authoritative provider evidence adjusts a charge once
as an append-only correction. Revoking a grant fences in-flight dispatch leases:
their transport status stays revoking until the lease expires or is
acknowledged, then becomes revoked; settlement of the original liability
remains charged to its original account set and is never transferred.
Operators recover a stuck backlog by confirming the reconciler is scheduled and
inspecting reconciliation-obligation counts, never by clearing holds manually.
Sharing-pool operation¶
Before publishing a binding, preview its selected ranges, snapshot/dynamic mode, independently shared facets, overlapping accounts, priority conflicts and active-range effect. Confirm authority across every included event or group. All-ranges means one deployment, not all customers. Empty collections must never expand into all ranges. Show shared-only pool exhaustion risk and offer per-range limits; do not silently partition a pool equally.
Changing a selector or routing revision preserves stable financial accounts, spent/reserved balances and original request liabilities. Publish with a revision comparison and fresh membership/authority checks. Revoke the old affected grant revision before reassessment; dynamic additions receive access only after admission. Monitor membership freshness, conflicts, invalidation lag, common-account contention and orphaned draw references without exposing group rosters in global telemetry.
Drain a pool by stopping new admissions and fencing member grants, then wait for active transports and unresolved provider outcomes under the ordinary reconciliation rules. Cleanup retains tombstones and original account vectors; deleting a group or binding cannot refund spent/unknown charges. Show which other bindings still reference the pool before changing provider identity. Revoking one range should leave its neighbors usable; disabling a shared provider identity is a separate, explicitly wider emergency action.
Restore qualification includes overlapping bindings, pinned shared assignments, membership revisions and unknown requests charged to old pools. Reconstruct authority before enabling admission; never recover balances by summing overlapping constraint accounts as if each were a separate provider invoice. See the sharing contract for the selector and allocation rules.
Observability and retention¶
Expose bounded metrics for admissions/denials by reason, active transports, reserved/spent units, unknown charges, lease age, revocation latency, allocation skew, provider throttles, price/observation age, refresh failures, control latency, audit backlog and cleanup residuals. Use bounded catalog IDs for metric labels; range/subject/request IDs belong in access-controlled audit/search records, not high-cardinality metrics. Prompts, responses, tool arguments, tokens and raw provider errors are forbidden in all telemetry.
Engine commits decision and accounting audit intent before dispatch. If that durable write fails, deny the call. External audit archival failure may continue only while the existing durable outbox is healthy and within its operator-set backlog bound; alert on lag and stop new model admission when that bound is exceeded. Do not create a broker-local best-effort audit store.
Initial deployment defaults: retain credential hashes only until expiry plus 24 hours for bounded abuse investigation; keyed retry fingerprints/request detail for 24 hours after settlement; body-free accounting/audit and evidence for 90 days unless the deployment's approved policy requires otherwise. Never expire unresolved charges, cleanup references or obligations merely because the normal retention period elapsed. Aggregate retained costs before deleting individual detail. Restrict exports to authorized operators; log their access. Document and test backup retention and deletion separately.
Rollout and legacy migration¶
- Inventory: identify all live range generations, stored full Vertex secret/key references, shared-key source consumers, AWS role consumers, model projects and effective IAM. Inventory metadata/references only; do not copy credential values into reports. Reconcile #1446 with the selected scenario so a new guest-key path is not introduced in parallel.
- Expand disabled: ship additive Engine schema, policy catalog, broker, identity/egress and operation projection behind deployment configuration. Existing ranges continue their recorded path. No new model grant is issued until the broker prerequisites and strict admission checks pass.
- Validate: run synthetic transport/accounting tests, effective IAM and root-context network negatives, token rotation/revoke and no-body leakage tests on disposable ranges. #2083's secret creation boundary must be proven for any remaining bootstrap/legacy-secret operations.
- Canary: create a fresh GCE range from the selected released image and scenario. Bootstrap broker-only client access, prove useful tool/model interaction, then pause/reset/revoke/destroy and independently inventory residuals. Charge probes to a separate bounded qualification account.
- Bounded cohort: explicitly enable the approved event/profile and measured range limit. Compare local reserved/used totals to provider observations, exercise overload and failure, and publish sanitized evidence.
- Drain legacy: move participants through fresh admitted generations; do not silently replace credentials in running guests. Coordinate shared GSA revocation across every consumer. Remove dynamic Vertex key creation and host secret-reader grants from the broker path; preserve original reference deletion until every old generation is retired.
- Contract: after no live legacy references and no effective legacy invocation, remove obsolete config and grants under the migration issue. #681's GCP readiness cannot rely on a disabled secret with a still-working previously minted provider token. Test that token's denial and record the measured window. AWS legacy cleanup remains independently scoped.
Do not rename a proposed ADR into a claim that live migration is complete.
Keep operator-visible legacy, broker-canary, broker-qualified,
unavailable and cleanup-pending deployment evidence/projections without
inventing new range lifecycle states.
Failure and incident procedures¶
| Trigger | Immediate action | Recovery/evidence |
|---|---|---|
| Provider 429/503 or quota exhausted | Stop admission to an unhealthy shard according to policy; return safe retry guidance. | Reassess fresh quota/health; explicitly reallocate only within approved model/region/price policy. No blind retry or cross-cloud fallback. |
| Engine/PostgreSQL unavailable | Reject new calls and refresh; terminate streams when continuation lease expires. | Recover service and reconcile unknown dispatch/settlement before new grants. |
| Broker process/node loss | Connection fails; keep request reservation and unknown outcome. | Replacement replica serves new admitted requests, not replay of the lost request; reconcile original provider reference. |
| Suspected participant-token theft | Revoke its grant epoch, fence streams and re-enroll after current authority checks. | Prove old token fails from its original subnet and another range; inspect body-free usage. |
| Suspected broker/provider identity compromise | Disable admission and drain the affected shards; deployment operator disables provider identity/permissions. | Measure effective denial with a pre-existing token, rotate/rebuild, reconcile spend and assess the entire affected shard inventory. |
| Budget/price mismatch or unknown billable feature | Disable that alias/feature and retain conservative reservations. | Review prices/usage adapter, publish a new revision and rerun billing-bound tests. |
| Secret/identity API outage | Fail new credential-dependent effects; no same-project/shared-key fallback. | Restore prerequisites; clean up by original references and re-enroll only through authoritative services. |
| Audit backlog exceeds bound | Stop new model admission and page operator. | Restore protected sink, verify durable backlog drains, then resume. |
| Slow clients or overload | Enforce request/concurrency/idle limits and bounded buffers; reject excess work. | Measure pressure and scale only within the qualified provider/database envelope. |
Rollback first disables new grants and drains admitted requests. Keep the expanded schema/readers and cleanup compatibility until all new references can be handled by the previous version. Roll back images/config only to a version that understands live grants, request records and cleanup obligations. If no compatible version exists, keep model access disabled and repair the forward release. Never restore direct guest credentials as a recovery step.
For database restore, hold the broker and enrollment route disabled before the restored application starts. Rotate a deployment-level admission epoch outside the restored database using the existing deployment secret/config authority, invalidate every restored grant, and conservatively freeze affected budgets. Reconcile the backup's request/accounting cutoff with protected audit and provider observations. Unknown missing charges consume remaining budget until an authorized conservative adjustment is recorded. Fresh enrollment uses the new admission epoch; old guest tokens cannot regain authority.
Qualification evidence¶
Allocation evidence¶
The M03 allocation boundary
persists quota commitments and pending grants before dispatch. Operators can
use model-access-policy/v2 to declare explicit provider-pool shard membership;
v1 remains readable without changing existing snapshot digests. A provider-pool
restriction without that mapping denies allocation.
Admission requires fresh, catalog-bound quota observations and complete owner authority. Observation collection must run outside database transactions. Missing observations deny required model use; enabling the catalog alone does not fabricate provider capacity. Pending grants cannot authorize broker calls. Local PostgreSQL and integration tests establish transaction behavior, not live provider availability or billing correctness.
Lifecycle refresh reads the durable, package-bound preparation snapshot. A catalog revision changes its identity even when scenario demand is unchanged; operators must publish matching fresh observations before renewal. Removing or disabling policy revokes any existing optional preparation and records visible absence rather than reusing its old catalog. Local development launches drain the production-shaped outbox after commit, so a rollback must produce neither a launch row nor a subprocess.
An initial optional launch while policy is unavailable is also durable: the operation records a catalog-free absence and cannot gain access on replay after policy is enabled. Routine Range status updates refresh selector evidence but do not revoke generation-stable launch authority. Owner, workspace, destroy and delete mutations remain revoking events and require fresh preparation.
Release evidence¶
The release bundle records repository/image/pack/client/SDK/chart digests, configuration and policy/price revisions, provider/project/region references under protected access, measured cohort/limits, effective IAM/network probe results, allocation/rate/spend contention results, first/last stream timestamps around revoke, cancellation/unknown-cost disposition, cleanup inventory and restore results. It contains no credential or prompt payload.
Prove useful participant model/tool use, multiple GCP projects and aliases, budget exhaustion, stale generation, wrong subject, expired grant, token refresh, worker loss, denied management/peer/metadata access, unavailable dependencies, price expiry, event stop and full teardown. Positive controls must show that allowed operations worked before negative results are credited. Independent provider readback is required; a script's success exit or a mock is not cloud evidence. Feed the qualified GCP slice to #2091 and keep each future provider/tool profile's evidence separate.
Broker execution configuration¶
GCP deployments configure settings.model_broker_runtime separately from
settings.model_broker infrastructure. Execution settings contain a
provider_inventory (model-broker-providers/v1), fingerprint_secret_name,
fingerprint_key_version, and optionally fingerprint_previous_secret_name.
They contain only non-secret bindings and Kubernetes Secret names. They do not
enter Terraform state. The fingerprint Secret's key entry contains 32–64 random
bytes; the optional retained-key Secret's keys.json maps previous versions to
base64-encoded keys. Retain old versions across the request deduplication window;
missing versions reject retries instead of admitting duplicate paid requests.
Each inventory target binds an enabled catalog shard's ID, provider, region,
model and credential reference to an approved invocation principal and a
conservative context_window_tokens. Vertex targets also specify their project
and explicit token-count geography. Projection rejects principals that differ
from applied Terraform identities. Provider credentials are obtained by workload
identity and never supplied in this configuration.
Activation requires an enabled v3 catalog with input/output prices and a zero-cost
request component for token counting. The v3 example
is synthetic and disabled: replace its model, quota, price and identity values
with approved deployment data before use. Infrastructure can remain installed
with model access disabled; both executable Deployments then have zero replicas.
The broker and provisioner use different authenticated control audiences. Only
provisioner Jobs may reach the enrollment control endpoint alongside the broker;
plugin workers receive neither control authority nor enrollment credentials.
Local rendering and transport tests do not establish live model/client compatibility. AWS deployment wiring and guest delivery must be qualified before advertising operational support on either cloud.
settings.model_broker_runtime.guest_trust_ca_pem supplies the public PEM CA used
by the enrollment control and guest broker listeners. It must match the existing
TLS trust ConfigMap. The deploy renderers encode this public certificate and
fixed private HTTPS coordinates in the runtime ConfigMap, which the provisioner
Job admission policy matches exactly. No CA private key belongs in root config.
Leaving this value empty prevents mapped guest enrollment before cloud mutation.
The broker/control TLS Secrets and public trust ConfigMap remain deployment-owned.
Vertex invocation and usage adapter (M07)¶
Issue #2124 hardens the existing broker Vertex path into a qualified invocation and usage adapter. ADR-059, ADR-060, ADR-061 and the Vertex adapter preflight retain their authority. Live qualification evidence is owned by #2334; local rendering, transport tests and a successful count are not live invocation proof.
Runtime behavior¶
- Count-based pre-dispatch charge bound. A paid request against a counting
provider is admitted on a free count bound (rate and concurrency are held, zero
spend) so a completed retry deduplicates before the provider is counted. After
the provider-proven count, the broker commits the input reservation to that
proven count and the output reservation to the request's
max_tokens, then dispatches the paid call. Routine small prompts are no longer denied because the reservation assumed the entire context window; hard spend, input and output ceilings and the immutable price schedule still apply. A provider with no free count endpoint keeps the conservative full-context reservation. - Bounded token reuse. Impersonated Vertex access tokens are cached only in broker memory, keyed by the complete approved target (authentication, principal, project, credential reference), with single-flight refresh and early expiry. A stored-credential revision keys a new entry. Tokens never cross the process or enter logs, exceptions, responses or accounting.
- Normalized failure and transport bounds. Non-200 provider statuses map to a bounded participant category without reflecting provider bodies, headers or request ids: 429 preserves client backoff, malformed-request statuses stay client-visible, and authentication or server failures surface as unavailable. The broker performs no provider retry, redirect or fallback, and an ambiguous dispatch retains a conservative liability rather than replaying a billable call. Upstream connect, write, pool and stream-idle limits are bounded separately under the absolute request/grant deadline so a useful streamed response is not cut at a short read while an idle upstream still cannot stall a worker slot.
Enablement prerequisites per target¶
Before enabling a Vertex target, record and validate: an enabled v3 catalog with
input and output prices plus a zero-cost request component for counting; the
approved project, inference region, explicit token-count geography (never
global), pinned publisher model and version, and an invocation principal that
belongs to the approved project; the real quota-pool identity, shared by every
model version or service account that draws on the same provider pool (identity
never multiplies capacity); and the current price, geography and retention
prerequisites. Provider credentials are obtained by keyless workload
impersonation and never supplied in configuration.
Installed client-settings contract¶
The pinned released client's installed configuration is the default that must let
a provisioned range work with no participant-side repair: it emits only the
qualified Anthropic Messages subset the broker admits (pinned anthropic-version,
no unqualified anthropic-beta or other cost-changing default features). An
unsupported cost-changing feature fails before a billable effect rather than being
silently stripped; a feature is admitted only when its protocol, provider
behavior, billing bound and usage settlement are all qualified. The exact client
version, installed settings and the sanitized real wire corpus that prove this
default are captured under #2334;
#2125 installs the same
pin and settings into guests without introducing a second compatibility target.