Testing: Clean-Checkout Commands and Quality Metrics¶
This document is the contract for running the test suites and for the coverage
and warning policy that makes the reported numbers trustworthy (REV1 Q6/Q7,
issue #1529). CI (.github/workflows/_quality.yml) is the enforcement authority;
the commands here reproduce its posture from a clean checkout.
Command matrix¶
This matrix is the single list of verification commands. CI is the enforcement
authority. Unless a row names another workflow, the CI job is in
.github/workflows/_quality.yml. The paths job selects jobs for the changed
paths, so a pull request runs only the jobs that own its changes.
Run the lanes for the packages you change. make test runs every row marked
yes in the make test column. Do not also run those child targets, because
that runs the same suites twice. Each make target syncs pinned dependencies
(uv sync or npm ci) and sets the CI posture. make help lists all targets.
Base prerequisites: uv and Node 20.19+ with npm. The prerequisites column lists
only additional tools or services.
Unit and package lanes¶
| Lane | Local command | Prerequisites | CI job | make test |
|---|---|---|---|---|
| Platform (SQLite, local only) | make test-platform |
None | None; CI runs the PostgreSQL lane | yes |
| Adapter SDK and packaging | make test-adapter-sdk |
None | Tests (adapter SDK) | yes |
| Engine provisioner | make test-provisioner |
None | Tests (provisioner) | yes |
| Packer helpers | make test-packer |
None; Packer cases skip without Packer | Tests (packer) | yes |
| Installation CLI | make test-installation |
None | Tests (installation) | yes |
| Bootstrap scripts | make test-bootstrap |
None (unit selection) | Tests (bootstrap) | yes |
| Layer-import checker | make test-check-layer-imports |
None | Tests (check_layer_imports) | yes |
| Helm chart contracts | make test-charts |
Helm | Tests (charts) | yes |
| Repository guard | make test-adr-guard |
None | Tests (adr_guard) | yes |
| GCP scripts | uv run pytest tests/ in scripts/gcp |
None (unit selection) | Tests (gcp scripts) | no |
| OpenVPN | uv run pytest tests/ in shifter/engine/openvpn |
None | Tests (openvpn) | no |
| Ops helper scripts | uv run pytest tests/ in each scripts/<helper> |
None | Tests (<helper>) |
no |
| Stack-smoke helpers | python3 -m unittest discover -s scripts/stack-smoke -p 'test_*.py' |
None | Lint and security (stack smoke) | no |
| UAT harness units | uv run pytest tests/ -q in each uat/<harness> |
None | Tests (<harness>) |
no |
| Terraform contracts | terraform init -backend=false, then terraform test, in the suite directory |
Terraform | Contract test (<suite>) |
no |
The ops helpers are handle_sd_replacement, check_rds_pending_modifications
and assert_portal_inspection. The UAT harnesses are event-load-harness and
range-functional-smoke. Their unit tests do not contact a deployment.
CI also runs the integration marker for bootstrap and GCP scripts. The local
commands select only unit tests for those packages. See
external tool integration tests.
Service and image lanes¶
| Lane | Local command | Prerequisites | CI job |
|---|---|---|---|
| Platform (PostgreSQL, coverage floor) | make test-platform-postgres |
PostgreSQL on :5432 |
Tests (shifter_platform, PostgreSQL n/6), Coverage (shifter_platform) |
| Platform Redis channel layer | make test-platform-redis |
Redis on :6379 |
Tests (shifter_platform, Redis channel layer) |
| OpenFGA authorization | ./scripts/run_openfga_integration.sh in shifter/shifter_platform |
Docker | Tests (OpenFGA 1.20.0, PostgreSQL) |
| Subnet migration proof | uv run pytest engine/tests/test_subnet_allocation_migrations_postgres.py in shifter/shifter_platform |
Docker | Tests (migration proof) |
| Guacamole authority | mvn verify in shifter/engine/guacamole/authority-extension, then the CI job's integration tests |
Maven, Docker | Tests and compilation (Guacamole authority) |
| guacd image | The CI job's image build, then bash shifter/engine/guacamole/test-entrypoint.sh |
Docker | Tests (guacd image) |
| Built-image stack smoke | bash scripts/stack-smoke/stack_smoke.sh |
Docker | Stack smoke (built image) |
| GCP deployer image | The CI job's fixture release manifest and image build, then docker run --rm bigrae-deployer:ci gcp-deploy --help and its Trivy scan |
Docker | Tests (GCP deployer image) |
The PostgreSQL and OpenFGA lanes run some of the same tests. They are not duplicates: each lane proves database or authorization behavior that the other lane cannot prove. The SQLite lane is a fast local lane; CI does not run it.
Frontend and browser lanes¶
| Lane | Local command | Prerequisites | CI job | make test |
|---|---|---|---|---|
| Legacy platform JavaScript (Jest) | make test-js |
None | Tests JavaScript (shifter_platform) | yes |
| React SPA lint, types, Vitest, build | make test-spa |
None | SPA (shifter_platform frontend) | yes |
| API contract and generated client | The command below, in shifter/shifter_platform |
None | API contract (shifter_platform) | no |
| Authenticated SPA journeys | make test-platform-e2e |
Chromium (the target installs it) | SPA E2E (shifter_platform frontend) | no |
| Browser accessibility | make test-platform-a11y |
Chromium (the target installs it) | Accessibility (deploy.yml) |
no |
The API contract check needs the test posture, as in CI. The secret key is a synthetic, non-secret value:
TESTING=1 TEST_DB_BACKEND=sqlite DJANGO_DEBUG=true \
DJANGO_SECRET_KEY=shifter-local-make-test-key \
uv run python manage.py api_contract --check
The two browser targets build the SPA and use SQLite. CI uses PostgreSQL. Each
target deletes shifter/shifter_platform/db.sqlite3. Do not run them at the same
time or against an active development server. Both lanes fail on a Content
Security Policy violation. See browser security policy.
MCP lanes¶
| Lane | Local command | Prerequisites | CI job | make test |
|---|---|---|---|---|
| MCP server packages | make test-mcp |
None | Lint JavaScript (mcp/<package>), Tests JavaScript (mcp/<package>) |
yes |
make test-mcp runs npm ci, npm run lint and npm test in each package
under mcp/. Lint allows zero warnings.
Policy and static checks¶
| Check | Local command | CI job |
|---|---|---|
| Architecture guard, imports, whitespace, changed prose | make policy |
Architecture (adr conformance), Architecture (import linter), Docs prose (changed Markdown) in vale.yml |
| Commit hooks | pre-commit run |
None; the hooks run locally before each commit |
| Workflows, Terraform, Kubernetes | The native commands in AGENTS.md | Lint (GitHub Actions), Lint (terraform), Lint and security (Helm chart), Lint (k8s schemas) |
make policy runs the ADR guard once. Do not also run the configured lint
command (adr_guard.py --all --level ci) in the same pass. Vale checks only the
committed Markdown changes against origin/dev.
Live qualification¶
These checks run against a deployed tenant. They need operator authority and are not pull-request gates. A pass proves only the scope that its runbook states.
| Check | Command or entrypoint | Runbook or status |
|---|---|---|
| Post-deploy range smoke | manage.py run_post_deploy_smoke, run by the deploy workflows |
Advisory: a failure opens an issue and does not block the deploy |
| Participant terminal and Guacamole session | uat/range-functional-smoke |
Range functional smoke |
| Event load envelope | event-load-harness |
GCP event capacity |
| Range escape boundaries | manage.py run_range_escape_validation |
Range escape validation |
| Model broker transport | scripts/gcp/probe_model_broker.py |
Model broker probes |
| Range audit telemetry | scripts/gcp/verify_range_audit_telemetry.py |
Range audit telemetry |
Range readiness or an open port does not prove a working terminal session. An accepted destroy request does not prove that teardown completed.
You can also run a package directly (cd <pkg> && uv run pytest) from a clean
checkout. The settings set the platform test posture (DEBUG, SQLite backend,
local provisioner off) through config._runtime_env.IS_TEST_RUN; CI does not
inject it.
CI platform lane¶
CI runs the platform suite one time, against PostgreSQL. pytest-split divides
the suite into six parallel shards. It uses the least_duration algorithm with
no durations file, so it deals the tests out to the shards in turn. Each xdist
worker clones its test database from one migrated template database. The
Coverage (shifter_platform) job then does these steps:
- It combines the coverage data from all shards.
- It enforces the whole-package coverage floor.
- It makes sure that at least one
postgres-marked test ran in a shard (scripts/check_postgres_lane_evidence.py).
A skipped postgres-marked test fails its shard. Keep each test fast. One slow
test makes its shard slow, because the shards do not use recorded durations.
Test posture (why a clean run matches CI)¶
config._runtime_env.IS_TEST_RUN is true when TESTING=1 or pytest is the entry
point. Under it the settings default to the test posture so a bare uv run pytest
does not inherit the production HTTPS posture a fresh checkout would otherwise get:
DEBUGdefaults toTrue(config/settings.py). An explicitDJANGO_DEBUGstill wins; production (IS_TEST_RUNfalse) is unchanged and defaults toFalse.LOCAL_PROVISIONERis forced off (config/_cloud.py) so range-lifecycle tests never shell out to a real provisioner subprocess (which leaks). A test that exercises that path setssettings.LOCAL_PROVISIONERvia the pytest-djangosettingsfixture.TEST_DB_BACKENDdefaults tosqlite(config/_database_settings.py).
These are covered by tests/platform/test_clean_checkout_posture.py.
Keep cloud test doubles within the provider adapter boundary. The tenant-pack and event-content fixtures select AWS and patch its storage constructor. They keep the shared storage factory and content resolver intact, so a module imported during a fixture cannot retain a temporary replacement after teardown. Content tests exercise the real parser and digest checks against synthetic stored bytes. When fixing an order-dependent failure, run the suspected predecessor and failing case together in one process. Also check that the next operation uses its own adapter after the patch ends. This follows Python's guidance on where to patch.
Coverage: two complementary floors¶
Production coverage measures owned code only. Each publishing package sets
[tool.coverage.run] source and omits the whole test tree, conftest hooks, and
migrations in its pyproject.toml, so tests can no longer cover themselves (the
platform metric was 94% self-covering, ~88% production-only). CI runs pytest
--cov (no path) so package config, not the command line, owns the measured set.
Absolute per-package floor, enforced by [tool.coverage.report] fail_under
in each package. Floors are per-package (a repository aggregate would let a
high-coverage package mask a regression in a smaller one). Each floor sits one
point under the measured production-only baseline to absorb run-to-run variance;
floors only stay level or rise, and a real reduction requires a reviewed,
time-bounded exception.
| Package | Production-only baseline | fail_under |
|---|---|---|
shifter/shifter_platform |
88% | 87 |
shifter/engine/provisioner |
80% | 79 |
shifter/installation |
97% | 96 |
scripts/bootstrap |
86% | 85 |
scripts/check_layer_imports |
96% | 95 |
shifter/packer |
76% | 75 |
SPA (frontend) floor. The Django-hosted React SPA
(shifter/shifter_platform/frontend) enforces its own absolute floors in
vite.config.ts (test.coverage.thresholds for statements, branches,
functions, and lines), each set one point under the measured baseline like the
Python packages so the floor absorbs run-to-run variance and only rises. The
measured source set is src/**/*.{ts,tsx} excluding tests, test support, and
the generated src/api/schema.d.ts; the first-party entrypoint and composition
root stay measured. The SPA's Vitest LCOV (frontend/coverage/lcov.info) is
uploaded by the SPA (shifter_platform frontend) CI job and restored into the
sonarcloud job, so SPA changed lines are held to the same changed-code gate
below (#1526).
Changed-code floor, owned by the SonarCloud raes-strict quality gate,
which fails a PR when new_coverage < 80 (80% coverage on changed lines), plus
new_violations > 0, new duplicated-lines density > 3%, any new rating worse than
A, and new security hotspots not fully reviewed. PR analysis waits for the gate
(sonar.qualitygate.wait=true). The conditions are recorded in
sonar-project.properties; each owned Python package publishes its configured
coverage report for Sonar to consume.
The sonar-rules-changed-lines pre-commit hook finds most new_violations
before a push. It applies the Python rules that fail the gate most often to the
lines that changed against origin/dev. It does not run tests, so it does not
check new_coverage.
The released OpenFGA/PostgreSQL lane also publishes coverage-openfga.xml,
including the native application and policy SQL duty tests. Sonar consumes both
platform reports so service-only behavior contributes real coverage evidence.
This supplemental report has no whole-package floor because it exercises a
service-specific subset; the Coverage (shifter_platform) job still enforces the
unchanged 87% package floor, and Sonar still enforces 80% changed-code coverage. Run it locally
from shifter/shifter_platform with
./scripts/run_openfga_integration.sh --cov --cov-report=xml:coverage-openfga.xml --cov-fail-under=0.
Warning policy¶
Warning classification lives in shifter/shifter_platform/pyproject.toml
[tool.pytest.ini_options] filterwarnings, so a direct pytest and CI agree
regardless of test order or xdist worker.
ResourceWarningand unawaited-coroutineRuntimeWarningare errors. They are correctness defects; fix the originating fixture/mock (close/await the resource) rather than downgrade the filter. Converting toignore, adding a sleep, forcing GC, or disabling xdist is not a fix.DeprecationWarningis an error by default. First-party deprecations, and any new or changed third-party deprecation, fail until fixed or explicitly baselined. Do not add a blanketignore::DeprecationWarning.- Third-party deprecation allowances are narrow and owned. Each is an
ignorefilter matched by exact message, with an adjacent owner and removal condition. The current set is thedjango-health-check(<5.0) class-based check deprecations, tracked for migration + removal in #1601. - Tests that intentionally assert a warning use
pytest.warnsat the behavior boundary; that is an assertion, not a baseline exception.
Authenticated SPA end-to-end (Playwright)¶
make test-platform-e2e runs the authenticated browser journeys against the
real Django-hosted built SPA (#1526). It is hermetic: it builds the SPA,
migrates and seeds a job-local database, and drives Playwright through normal
Django sessions and CSRF. The CI job shifter-platform-e2e in
.github/workflows/_quality.yml feeds the PR Gate. It runs the journeys
through scripts/run_openfga_integration.sh --browser, which starts a job-local
PostgreSQL and OpenFGA server, migrates, and seeds. The make target uses SQLite
locally.
The dev/test-only seed_e2e management command establishes the synthetic
actors, their accounts and their native policy grants; dev_login establishes
their sessions. The dev/test-only seed_e2e_ctf command runs after seed_e2e.
It adds one CTF event in the organizer's account, publishes the event's
authorization relationships, and releases inbox messages for the inbox
participant actor. The Playwright config's auth.setup.ts logs each actor in once and reuses its
storageState. No cloud, provisioner, IdP, or live-range dependency is used:
LOCAL_PROVISIONER/ENGINE_TASK_* stay unset, so a range launch enqueues
nothing and contacts no cloud.
Hermetic scope: login/auth-revocation, workspace lifecycle (create/rename/archive/restore), shared-audit history, authorization denial (client advisory and the authoritative API 403), CTF organizer/participant/threat-research role landings, and the participant inbox with stored and reflected XSS probes under the enforced CSP. Journeys that require a live provisioner (range readiness/terminal/cleanup, CTF range, Guacamole) are the deployed tier and are tracked separately, not run in the PR lane.
Authenticated Playwright traces are retained only on a local retry and are never
uploaded as CI artifacts (ADR-055-R7). Reproduce failures locally with
make test-platform-e2e.
Browser accessibility (Playwright + axe)¶
make test-platform-a11y runs @axe-core/playwright over the ADR-055 surface
matrix (frontend/e2e/a11y/) against the same hermetic stack, with the WCAG 2.2
A/AA tag set. The Accessibility job in .github/workflows/deploy.yml runs it on
every pull request as a direct PR Gate dependency (it does not inherit the
frontend path skip, per ADR-055-R2), against a job-local PostgreSQL.
The debt baseline is an exact set of privacy-safe fingerprints
(frontend/e2e/a11y/baseline.json): the scan fails on any new or resolved
finding, and the accessibility-baseline adr_guard check enforces that the
committed baseline may only shrink versus the trusted base branch (additions
require an exact-fingerprint waiver in docs/adr/exceptions.yaml naming ADR-055,
per R4/R6). The matrix reconciles fail-closed against the SPA route table
(src/test/a11y-surface-reconcile.test.ts); a new route group with no coverage
or exclusion fails. Nightly multi-browser runs, deployed-target scans, the
WCAG-EM manual audit, and the release-evidence gate are the remaining ADR-055
tail tracked in #2117.
Browser security policy¶
The portal enforces the ADR-036 Content Security Policy. The browser lanes record
each securitypolicyviolation event through frontend/e2e/support/csp.ts. A
journey, accessibility scan or live Django page fails on any violation.
e2e/journeys/csp.spec.ts checks the enforced header and blocked inline script
and handlers. It also checks that same-origin WebSockets connect and off-origin
connections are blocked. e2e/journeys/django-pages.spec.ts covers the live
Django-rendered pages. These tests use Chromium only. The deployed identity
provider, MFA and range terminal paths are outside the hermetic lanes; #2117
owns that tier. RB024 is the
operator procedure for a blocked feature.
External tool integration tests¶
The GCP scripts, bootstrap, adapter SDK and Packer packages register the
integration pytest marker for checks that require external rendering/build
commands or isolated package/TLS environments. Use uv run pytest -m "not integration"
from a package directory for its tool-independent unit selection, or
uv run pytest -m integration to run the external-tool checks explicitly.
GCP scripts and bootstrap default to the unit selection; their CI jobs explicitly
select both with -m "integration or not integration". SDK and Packer default
runs retain both selections. No regression checks are removed from CI.
The bootstrap and GCP script CI jobs install the same digest-verified Helm build; the GCP script job also installs the pinned kubectl client. Bootstrap verifies its OpenSSL prerequisite. SDK wheel conformance requires uv and the runtime extra's cached dependencies; guest TLS integration requires OpenSSL. Packer validation requires Packer and its plugins. These checks do not deploy cloud resources.
shifter/packer/tests/test_scripts.sh and the TestScriptStructureLint and
TestScriptContentLint classes are lint-only checks. Their success does not prove
provisioning behavior or qualify an image. The script execution suites use
controlled command shims; built-image behavior still requires image qualification.