Testing

apomesh separates its tests into tiers by what each one asserts and what it costs to run. Most tiers run offline with no credentials; the ones that touch Docker or real provider endpoints are gated so a plain cargo nextest run stays fast and deterministic. This page is a how-to: what to run while you iterate, the command per tier, and how to write a test that fits the substrate.

The fast loop

While you iterate, scope the runner to the crate you are touching instead of the whole workspace:

cargo nextest run -p apomesh-substrate            # one crate
cargo nextest run -p apomesh-substrate supervisor # + substring filter on test name

nextest treats trailing arguments as a substring filter over test names, so the second form runs only tests whose name contains supervisor. When the change is ready, run the full local gate (below) — the same sequence CI runs — before opening a PR.

The canonical runner

The canonical test runner is cargo nextest, paired with cargo test --doc because nextest does not run doc tests:

cargo nextest run --workspace      # unit + integration (Tier 1 + Tier 2)
cargo test --workspace --doc       # doc tests

Install once with cargo install cargo-nextest --locked. nextest gives per-test process isolation (catching global-state leaks cargo test’s in-process scheduler hides), real-time progress, and a ci profile with one retry and JUnit output.

The runner is configured in .config/nextest.toml. Two details worth knowing:

• Sandbox setup script. The Linux (bubblewrap_linux) and macOS (sandbox_exec_macos) sandbox integration tests dispatch through a real apomesh-tool-runner subprocess built with the test-tools feature. A setup script (build-tool-runner) compiles that runner once before the parallel phase — without it, process-per-test isolation makes every sandbox test race the compiler on cargo’s build locks. The script’s filter fires only when a sandbox test is actually selected, so a cassette-only run skips it.
• Slow timeout. Tier 2 cassette tests exceed one second by design (a mock-server start plus a dispatch round-trip), so slow tests are flagged, not killed.

Note

The two sandbox subprocess tests depend on that prebuilt test-tools runner, so run them under nextest — a bare cargo test will not prebuild it. bubblewrap_linux is #[cfg(target_os = "linux")]-gated; macOS compiles it out and runs sandbox_exec_macos instead.

The test tiers

Tier	Covers	When	Command
Tier 1 — unit	substrate-internal functions on hand-authored inputs	every workspace run	cargo nextest run --workspace
Tier 2 — cassette replay	the full pipeline against frozen, recorded provider responses	default CI on every PR	cargo nextest run --workspace
Doc tests	examples embedded in /// docs stay correct	separate step	cargo test --workspace --doc
testcontainers	the Postgres / Redis adapters against a real database	every PR, in the Docker-equipped CI job	cargo nextest run -p apomesh-state-store-postgres --run-ignored=all
Tier 3 — cassette refresh	whether a live provider’s wire shape drifted from the baseline	monthly GitHub Action	cargo run -p cassette-refresh -- check
Tier 4 — live-provider	the full pipeline against the live provider wire	on-demand GitHub Action	cargo nextest run -p apomesh-substrate --features live --test live

Cassette replay (deterministic, no keys)

apomesh has two record-then-replay cassette systems, one per side of the wire. Both reproduce byte-for-byte with no API keys and no network — see Replay and history for the concept.

Substrate side (Tier 2). A VCR records at the LLMProvider trait, not the HTTP wire: RecordingProvider wraps any provider and captures each request/response pair; ReplayProvider serves them back, matching each request to a recorded response by a stable SHA-256 digest of its identifying fields (a missing match is a hard, loud miss — replay never falls back to a live call). This is what lets the deep-research eval harness and the orchestrator’s compute-matched bake-off record once and replay deterministically.

SDK side. The Python SDK replays recorded Envelope sequences against an in-process StubDaemon rather than a live daemon: load_cassette_into_stub primes the stub from a committed JSON fixture under sdks/python/tests/cassettes/, and the tests assert the SDK’s ergonomic surface. APOMESH_CASSETTE_MODE selects the mode (replay is the default). To record a new one, run the auditable bootstrap script under sdks/python/scripts/ and commit the script edit alongside the regenerated fixture — the protocol is in the cassettes README.

testcontainers integration

The Postgres and Redis adapter tests (in crates/apomesh-state-store-postgres/tests/integration/) boot real containers via testcontainers — a pgvector Postgres image with migrations applied, and a Redis image for the composed HybridStateStore. They need a running Docker daemon, so they are marked #[ignore = "requires docker; run with --run-ignored=all"]; a Docker-less run skips them cleanly, and you opt back in explicitly:

# requires a running Docker daemon
cargo nextest run -p apomesh-state-store-postgres --run-ignored=all

This #[ignore] + --run-ignored=all pattern is how any Docker-dependent integration test is kept out of the default fast path.

CI runs them on every PR. The rust-integration job has a Docker daemon, so after the workspace run it opts these back in with a second, crate-scoped invocation:

cargo nextest run \
  -p apomesh-state-store-postgres \
  -p apomesh-state-store-redis \
  --run-ignored=all --profile ci

The scope is deliberate rather than a workspace-wide --run-ignored=all. The other ignored tests are the live-provider suites, which need real credentials and bill real money; a blanket flag would fire those too. Naming the two adapter crates keeps the Docker class and the credential class apart.

These adapters share their test bodies with the in-memory and SQLite stores through the state_store::contract conformance suite, so a behavioural drift on any one adapter lights up immediately — see Writing tests.

Cassette refresh (Tier 3)

cassette-refresh replays the recorded scenarios against the real endpoints to detect wire-shape drift — a distinct job from replay. It runs monthly (opening an issue on drift) and locally via cargo run -p cassette-refresh -- check; full CLI, manifest format, and drift-report reading are in docs/cassette-refresh.md.

Live-provider tests (Tier 4)

Tier 4 runs a real request through the substrate’s actual dispatch pipeline against a live provider endpoint, asserting the pipeline still works end to end (where Tier 3 only diffs bytes). It runs on demand via GitHub Actions workflow_dispatch, reading credentials from repo secrets. Locally:

cargo nextest run -p apomesh-substrate --features live --test live

Without credentials, every live test skips (printing a SKIP line) rather than failing — so this command is safe to run in a keyless checkout. The full workflow, credential registration, and cost budget are in docs/live-provider-tests.md.

The full local gate

Before opening a PR, run the same checks CI runs for the Rust workspace:

cargo fmt --all --check
cargo clippy --workspace --all-targets --all-features -- -D warnings
cargo nextest run --workspace
cargo test --workspace --doc

A plain cargo build / cargo nextest run (no -p, no --workspace) skips the Tauri GUI crate (apomesh-control-plane) — its webview dependency tree is a large compile cost irrelevant to substrate work, so CI and the release gate pass --workspace explicitly.

Per-stack gates

The SDKs and the control-plane webview have their own gates, run from their own directories.

Python SDK (from sdks/python):

uv sync --all-extras
uv run python scripts/regen_proto.py && git diff --exit-code apomesh/proto/  # stub-drift gate
uv run ruff check .
uv run ruff format --check .
uv run ty check
uv run pytest

The stub-drift step regenerates the proto stubs and fails if the committed output has drifted from the canonical .proto — the guard that keeps the generated surface honest.

UI control plane (from ui/control-plane): the desktop crate is checked with cargo fmt -p apomesh-control-plane --check, cargo clippy -p apomesh-control-plane, and cargo build -p apomesh-control-plane; the Svelte webview runs bun run format:check, bun run lint, bun run check, bun run test, and bun run build.

Continuous integration

CI is stack-aware. A detect job probes the checkout for each stack’s entry file (Cargo.toml, the SDK manifests, ui/control-plane/package.json) and downstream jobs gate on those outputs, so the matrix activates automatically as stacks land. Rust runs as parallel fmt+clippy+unit and integration jobs; the SDKs and the control plane each run their per-stack gate.

Markdown, docs/, .claude/, and ai_docs/ paths are ignored, so a docs-only change triggers no workspace CI. Three surfaces stand apart:

• Docsite — its own workflow (ci-docsite.yml), triggered only on docsite/**, running bun run build (the coverage check rides the build).
• cassette-refresh — monthly cron, opens an issue on drift.
• live-provider tests — workflow_dispatch only, on demand.

Compile-time canaries

Some of the substrate’s strongest tests are not test functions — they are compile-time exhaustiveness checks. A wildcard-free match over a closed enum, a const _: never assertion in the TypeScript mirror, and the worker’s payload_dispatch_canary all fail the build the moment a new closed-enum variant lands unhandled on that surface. Adding a Payload variant that reaches the wire but not a consumer’s projection cannot compile — the drift is a build error at authoring time instead of a mystery event in production.

Writing tests

Two conventions carry most of the substrate’s assurance:

Adapter conformance. When a trait has multiple deployment-bound adapters (the StateStore family), write the behavioural assertion once in a shared contract module and run it against every adapter. state_store::contract is the model: one set of assertions, run against in-memory, SQLite, Postgres, and the composed Hybrid store, so no adapter can silently diverge.

Round-trip canaries for closed enums. A closed enum mirrored across the wire is covered by a round-trip canary driven off an exhaustive list, not a hand-maintained switch: the Tauri payload_variant_round_trips test iterates ALL_PAYLOAD_VARIANTS, and the Python test_round_trip.py round-trips every wire-mappable type. Adding a variant without extending the driver breaks the build or the test — which is the point, and the contract-parity rule is the discipline it enforces.

Next steps

• Find the crate you are testing: Codebase orientation.
• Working on the docs build gate instead? See Working on the docs.