Scenario Verification Plugins¶
BigRAE's scenario-verification package provides a neutral contract for discovering an installed verification plugin, running its checks through an injected transport, and rendering one redacted report. Core ships no per-scenario adapters, answer material, target topology, concrete runner, or plugin dependency.
Use this surface from a dedicated, least-privilege operator environment. It is not a Django app, service-startup hook, scenario loader, backend admission gate, or built-in scenario CLI.
Public v1 surface¶
Import public types from shared.scenario_verification, not its internal
modules.
| Surface | Public names | Purpose |
|---|---|---|
| Versions and limits | ENTRY_POINT_GROUP, API_VERSION, REPORT_SCHEMA_VERSION, MAX_OUTPUT_BYTES |
Fix the discovery namespace and version the plugin/report contracts; expose the runner output bound. |
| Plugin metadata | AdapterDeclaration, PluginDeclaration, AdapterCallable |
Frozen declarations returned by the selected plugin's zero-argument factory. |
| Adapter execution | Binding, AdapterContext, Runner, ExecResult, CancellationToken |
Namespaced opaque targets and the bounded argv-only transport available to adapter code. |
| Adapter verdicts | AdapterStatus, AdapterOutcome, CheckReason, equal_without_disclosure |
Limit adapters to pass/fail and compare values without returning or fingerprinting either operand. |
| Framework results | CheckStatus, CheckResult, VerificationReport |
Represent the closed pass/fail/blocked/error check results and one immutable aggregate report bound to installed metadata. |
| Discovery | InstalledPlugin, LoadedPlugin, PluginSelection, discover_plugins, load_plugin, PluginDiscoveryError |
Enumerate metadata without loading code, then load and bind only the exact reviewed selection. |
| Orchestration and rendering | run_verification, render_human, render_json, aggregate_exit_code |
Execute deterministically and render/exit from the same aggregate DTO. |
| Failure signals | RunnerExecutionError, VerificationCancelled, VerificationDeadlineExceeded, VerificationConfigurationError |
Classify runner faults and reject cancellation, budget exhaustion, or unsafe execution configuration without exposing raw messages. |
The dataclasses are frozen. Identifiers and reason codes are bounded lowercase identifiers; plugin and adapter ids and binding names are namespaced. Summaries are non-empty, bounded, and rejected if they contain unsafe control or escape characters.
API_VERSION versions the factory/declaration ABI. REPORT_SCHEMA_VERSION
versions the JSON evidence shape independently. A plugin with an unsupported
API version fails before any adapter runs.
Publish a plugin factory¶
An installed distribution registers exactly one or more factories in the fixed entry-point group:
[project.entry-points."shifter.scenario_verification.adapters"]
synthetic = "synthetic_verification:plugin"
The referenced callable takes no arguments and returns a
PluginDeclaration. This synthetic example deliberately contains no real
scenario id, target, topology, answer, or credential:
from shared.scenario_verification import (
API_VERSION,
AdapterContext,
AdapterDeclaration,
AdapterOutcome,
AdapterStatus,
CheckReason,
PluginDeclaration,
equal_without_disclosure,
)
def verify_alpha(context: AdapterContext) -> AdapterOutcome:
result = context.run(
"synthetic.primary",
("probe", "--format=json"),
timeout_seconds=10,
)
matched = result.exit_code == 0 and equal_without_disclosure(
result.stdout.strip(),
expected_value_from_plugin_memory(),
)
return AdapterOutcome(
AdapterStatus.PASS if matched else AdapterStatus.FAIL,
CheckReason.VERIFIED if matched else CheckReason.MISMATCH,
)
def plugin() -> PluginDeclaration:
return PluginDeclaration(
api_version=API_VERSION,
plugin_id="synthetic.pack",
plugin_version="1.0.0",
adapters=(
AdapterDeclaration(
adapter_id="checks.alpha",
summary="Synthetic availability check",
execute=verify_alpha,
),
),
)
The expected value in this example is plugin-owned data kept in memory. It is
never put in a declaration, binding, result summary, exception, log, or report.
equal_without_disclosure returns only a boolean and does not create a
deterministic hash.
An adapter may list other adapter ids in prerequisites. The framework
validates that every dependency exists, rejects cycles, and executes checks in
stable prerequisite order. Use prerequisites only for runtime conditions that
can be satisfied by another declared check. Do not use them as a backend
capability manifest or a substitute for missing coverage.
Select installed code explicitly¶
discover_plugins() reads only installed distribution and entry-point
metadata in ENTRY_POINT_GROUP; it does not import candidates. Select by exact
distribution name, installed version, and entry-point name:
from shared.scenario_verification import (
PluginSelection,
discover_plugins,
load_plugin,
)
candidates = discover_plugins()
loaded_plugin = load_plugin(
candidates,
PluginSelection(
distribution="synthetic-verification",
version="1.0.0",
entry_point="synthetic",
),
)
The distribution should be reviewed and pinned by version and artifact digest
when the operator environment is built. Installation plus exact selection is
the code-loading authorization boundary. The framework never installs a
package, follows a URL, scans a directory or namespace, changes sys.path, or
accepts an arbitrary module path. When exactly one candidate is installed,
load_plugin(candidates) may select it; explicit selection is preferred for
repeatable operator evidence.
Discovery, factory, and declaration failures raise PluginDiscoveryError.
Diagnostics identify bounded distribution/entry-point metadata and the
exception class where useful, never the raw exception message. These failures
occur before a VerificationReport exists; the operator entry point must catch
them and exit non-zero.
Inject a runner and bindings¶
The operator owns the concrete Runner. Its run method receives:
- an opaque
target_idresolved from a declared binding; - an argv tuple, never a shell command string;
- optional string stdin;
- an explicit timeout bounded by the remaining whole-run deadline.
It returns a bounded ExecResult(exit_code, stdout, stderr, duration_ms).
stdout and stderr are available to the selected adapter in memory but are
never report fields. The concrete runner must validate its target namespace,
constrain output before buffering, avoid broad inherited environments, and
terminate or cancel remote work when the run budget expires. It raises
RunnerExecutionError for a transport fault and TimeoutError for a command
timeout; raw transport messages never enter the report.
Build Binding values from operator-controlled non-secret configuration:
from shared.scenario_verification import Binding
bindings = (
Binding(name="synthetic.primary", target_id="target-a"),
)
Bindings map logical names to opaque runner targets. They must not contain
credentials, provider objects, network details for reporting, settings,
environment maps, or scenario answer material. Adapter code calls
AdapterContext.run(binding_name, argv, stdin=..., timeout_seconds=...); it
cannot choose an undeclared target.
Run and interpret verification¶
run_verification(plugin, context, *, selected_adapter_ids=None) receives the
validated LoadedPlugin and an AdapterContext that already owns the injected
runner, bindings, whole-run deadline, cancellation signal, and monotonic clock.
The loaded wrapper binds the declaration to the exact installed distribution,
version, and entry point selected during discovery. The call returns one
immutable VerificationReport. An explicit selection is a tuple of adapter
ids; it must be non-empty and prerequisite-closed or validation fails before
adapter execution with VerificationConfigurationError. The optional
monotonic callable exists for deterministic duration tests; production callers
normally omit it.
import time
from shared.scenario_verification import (
AdapterContext,
aggregate_exit_code,
render_human,
render_json,
run_verification,
)
context = AdapterContext(
runner=operator_runner,
bindings=bindings,
deadline=time.monotonic() + 300,
cancellation=operator_cancellation_token,
)
report = run_verification(
loaded_plugin,
context,
selected_adapter_ids=("checks.alpha",),
)
human_output = render_human(report)
json_output = render_json(report)
exit_code = aggregate_exit_code(report)
The operator decides where the rendered strings go and exits with exit_code.
The framework does not write files or print on the caller's behalf.
Adapters return only:
AdapterOutcome(AdapterStatus.PASS, CheckReason.VERIFIED); orAdapterOutcome(AdapterStatus.FAIL, CheckReason.MISMATCH).
The framework maps execution into the closed report statuses:
| Status | Meaning |
|---|---|
pass |
The adapter ran and returned the verified verdict. |
fail |
The adapter ran and returned the mismatch verdict. |
blocked |
A declared prerequisite was unsatisfied, so the dependent adapter did not run. |
error |
Prerequisite execution, runner, timeout, cancellation, invalid-result, or adapter execution faulted after selection. |
The closed CheckReason values are verified, mismatch,
prerequisite_unsatisfied, prerequisite_error, adapter_error,
runner_error, timeout, cancelled, deadline_exceeded, and
invalid_result. Plugins cannot add free-form report reasons.
blocked is a non-success result. It is not the status for absent adapter
coverage, an unsupported API version, an admission/realizability failure, or an
exception. Every fail, blocked, or error makes
aggregate_exit_code(report) non-zero. Discovery and execution-configuration
exceptions happen before a report and must also produce a non-zero process
exit.
Render operator output only through render_human(report) or
render_json(report). Both consume the same DTO, so redaction and status
semantics cannot diverge between formats.
Report schema and redaction¶
The JSON report has exactly these top-level fields: schema_version,
selection, checks, summary, duration_ms, and exit_code. selection
contains distribution, distribution_version, entry_point, plugin_id,
and plugin_version, binding the report to both installed metadata and the
factory declaration. Each check contains adapter_id, status, reason, and
duration_ms; summary contains pass/fail/blocked/error/total counts. Treat
the rendered document as ephemeral operator evidence. The framework does not
persist it or write a database, audit event, or artifact record.
Neither report format includes:
- expected or produced values, flags, credentials, keys, or tokens;
- deterministic value hashes or fingerprints;
- argv, stdin, stdout, stderr, environment values, or raw evidence;
- exception messages or tracebacks;
- internal hostnames, addresses, provider payloads, or deployment settings.
Plugin-authored ids and declaration summaries are untrusted input. The contract validates and sanitizes them before execution; summaries are not copied into the report. Do not add a debug/raw-output report mode; inspect sensitive failures only inside the controlled plugin/runner process.
Boundaries¶
Scenario verification observes one realized run. It does not parse scenario SDL, declare required capabilities, select a backend, or change launchability. Scenario SDL remains demand, the backend manifest remains supply, and the runtime-target, ingest-compatibility, and realizability gates reconcile them before dispatch.
Challenge-board or scoring-system readback, provider topology, persistence, deployment configuration, service APIs, and platform lifecycle status are also outside this ABI. Keep any optional readback in explicit operator tooling, and pass only the minimal non-secret bindings and least-privilege runner needed by the selected adapters.