Full Application Test Matrix

`scripts/test-matrix.sh` RUNS every leg

Every stage executes its leg's real command and reports a real verdict. This has not always been true: stages 2, 3 and 4 used to check a precondition (dev stack up / stopped / scanners on PATH) and then write NOT-MEASURED … execution is a separate, future effort to the report and return 0. scripts/test-matrix.sh all therefore exited 0 having proven the eight Stage 1 checks and the leg table's own doc-sync — and nothing whatsoever about GPU scaling, diarization, lite mode, auth, PKI, fresh install or upgrade, while listing every one of them. A green checklist that measured none of that is worse than having no leg at all, and it is gone.

Five outcomes, and none of them is a placeholder:

Outcome	Meaning	Run exits
`PASS`	the leg ran and its own criteria held	`0`
`FAIL`	the leg ran and did not pass; the report names the log file	`1`
`SKIP`	the leg ran and reported in its own words that it could not measure this here — printed with that script's stated reason, counted, and repeated loudly at the end	`0`
`BLOCKED`	the leg found a precondition it could not meet	`3`
`ABORT`	the leg reported an operator abort (e.g. a declined `I UNDERSTAND` prompt)	`4`

A run with any SKIP prints A green matrix with skips is not a fully measured one. — the same discipline scripts/audit-tests.py applies to its DEFERRED count.

BLOCKED and ABORT exist because collapsing every non-zero into FAIL made "the operator said no" and "the rehearsal found a regression" look identical. gr_confirmation_gate in scripts/release-tests/lib/guardrails.sh now exits 4 (it used to gr_die, i.e. 1), and scripts/release/65-rehearse.sh preserves 3/4 instead of flattening them — so a rehearsal nobody agreed to run is no longer recorded as a rehearsal that failed.

Exit codes are the shared contract with scripts/release.sh: 0 pass, 1 gate failed, 2 misuse, 3 precondition unmet, 4 operator abort. Stage 3 legs additionally require --yes, because they rebuild images and rehearse real deployments for hours.

⚠️ This repo has two exit-code conventions, and each leg declares which one its command follows (the fifth field of test-matrix.sh's LEGS table, shown by --list):

standard — the contract above. Used by everything except the three smoke scripts.
smoke — 0 pass, 1 check failed, 4 NOT MEASURED. Used by gpu-scale-smoke.sh, diar-native-smoke.sh and lite-smoke.sh, whose own headers document it.

So exit 4 means "operator abort" in one and "not measured" in the other. Declaring it per leg is what lets the dispatcher report a smoke script's honest "there is no running diar-native container" as a SKIP with that reason, instead of misreading it as an abort. backend/tests/unit/test_test_matrix_execution.py fails if a leg's contract stops matching the script it wraps, if a leg names a path that does not exist, or if the placeholder ever returns.

Time budget follows from this being real: Stage 1 is minutes, Stage 2 is hours, Stage 3 is hours. That is the honest cost of the coverage, not a reason to defer it.

Coverage stance at a glance

Read this table before assuming a mode is covered — several gaps below were only found because someone assumed "the matrix covers it" without checking which stage actually asserts what. Each gap-hunt pass on this repo has found new ones; treat this table as a living record, not a finished checklist. "Real" means the leg drives the actual application path (upload, restore, login) and asserts on outcome, not just that a command exited 0.

Mode	Stage / leg	Coverage	Known gaps
Dev — baseline	Stage 2A	Real: upload, e2e, chat vs mock + real vLLM, all 3 auth IdPs	Leg 4 (real vLLM) is not reproducible on a single 12 GB GPU — the default model does not fit at any documented setting (#608); see Cycle 2A below for the numbers and the (unverified) `--enforce-eager` escape hatch
Dev — GPU scaling	Stage 2B	Real: N-worker topology, concurrent uploads, OOM check	#609: the Flower leg only proves the worker answered a `?refresh=1` broadcast, not that N workers are registered in the plain `/api/workers` snapshot — that endpoint is a one-shot cache taken at Flower's own process startup and is never refreshed on a timer, so an unrefreshed read can omit a healthy worker forever
Dev — diarization	Stage 2C	Real: diar-native default path. PyAnnote fallback coverage is partial — see gaps	`diar-native-smoke.sh` exits `4` (NOT MEASURED) when no sidecar container is found. `test-matrix.sh` now exits its own `EXIT_NOT_MEASURED` (5) rather than 0 whenever any leg reported NOT MEASURED, and `run-integration-tests.sh` treats it as a non-fatal skip only when the sidecar was not expected (`diar_native_sidecar_expected()`: engine.diarizer_backend resolves to native AND weights-or-token are configured) — when it WAS expected, NOT MEASURED is filed as a FAILURE, not a skip (issue #669). Of the four suites this leg leans on: `test_native_embedding_equivalence.py`'s core sidecar-vs-PyAnnote equivalence claim is now asserted against a frozen, committed reference vector (`tests/fixtures/embeddings/pyannote_v4_reference_vectors.json`) and needs no `pyannote.audio` import, so it survives PyAnnote's absence — one PyAnnote-requiring re-validation test remains inside it and is separately named. The `karpathy_10m.` PyAnnote fixture trio is still live (PyAnnote is not being removed on this branch) with a corrected, per-trio-verified regeneration procedure. `test_diarizer_lifecycle.py` is untouched by this pass and remains PyAnnote-pinned. `test_diarization_perf_gates.py` measures PyAnnote-on-A6000 timings only* — it does not gate diar-native's own performance at all, and its docstring now says so explicitly rather than implying native coverage; no native perf gate exists yet (issue #669)
Dev — lite/CPU	Stage 2D	Topology-only — proves no GPU worker/memory, uploads nothing	The pipeline itself (ASR/search/chat) is NOT exercised here — see the lite-mode rehearsal row below
Prod — fresh install	Stage 3	Real: full install against a built image	—
Prod — upgrade	Stage 3	Real: version upgrade path	—
Prod — backup/restore/rollback	Stage 3 (`test-upgrade.sh` phases 06b, 12–17, #598) — this leg now runs to completion across all 18 phases; two harness bugs that used to silently truncate it mid-run before ever reaching here are fixed (#617, #618)	Real: `opentranscribe.sh backup`/`restore` (issue #613 — the shipped production command; the rehearsal staged `opentr.sh` for this until #613, which was itself an invalid bare-`docker-compose` invocation outside a repo clone) and `update --rollback`, damage injected via the real API, restore asserted by content digest (not row count)	`backup --encrypt` (unattended gpg needs a passphrase file the CLI doesn't support); the in-app scheduled-backup system (`backup_service.py`) has real unit/API coverage but no end-to-end restore proof — see #604 for its one known remaining defect (gnupg missing from the backend image); MinIO/OpenSearch restore is not touched by the DB restore path; phases 15/17's before/after digest comparisons can still race an async DB write settling after the harness's synchronization point — for the `speaker` table, gender-attribute writes are closed by waiting on `attributes_predicted_at IS NOT NULL` (#617's fix), and LLM speaker-suggestion writes are closed by waiting on no pending/in-progress `speaker_identification` Task row plus 3 consecutive stable digest polls (#620 item 5, a best-effort settle for any other writer, not a guarantee); `media_file` content and `system_settings.embedding_normalization_done` are closed the same best-effort way — `dbs_wait_for_media_file_settled`/`dbs_wait_for_system_settings_settled` wait for each table's content digest to stop moving before phase 06b's before-snapshot and the shipped-vs-wrapper backup diff, rather than tracking every individual writer by name (#619)
Prod — lite-mode pipeline	Stage 3 (`test-lite-mode.sh`)	Real: full upload→ASR→search→chat against mocked cloud ASR + mocked LLM, no GPU/vendor key needed	Mock's per-request `?scenario=` override isn't reachable from `GladiaProvider` — the negative-path check restarts the mock container instead of driving it per-request
Prod — PKI/mTLS	Stage 3	Real: client-cert auth, cert-less request rejected at the TLS layer	Prod+nginx only by design — no dev-mode variant exists (Vite can't terminate mTLS)
Prod — lite/gpu-scale as deployment modes	—	Compose-validated only (Stage 1.6)	No separate prod runtime pass — deliberate scope decision, since prod images behave identically to dev images for these flags and Stage 2 already proves the runtime behavior
Offline / air-gapped	Stage 1.6	Config-validated only	No real network-namespaced offline install pass exists — do not read Stage 1.6 as proving offline mode works end to end
Image/release gates	Stage 4	Real: scan/build/publish/promote wiring	`scan`'s security-tooling check is warn-severity, not blocking — verify trivy/grype/syft are on `PATH` before relying on it

Pattern to watch for: nearly every gap this repo has found (#598–#604) was a script or feature that looked covered because something referenced it — a doc section, a RUN_* env var, a constant — but nothing exercised the actual failure path. When adding a new script or rehearsal leg, ask "what does this look like when it silently does nothing, or does the wrong thing and still exits 0?" and write that test first.

Stage 1 — Static / no live stack

~6-9 min. CI-safe: needs no GPU and no running stack.

#	Command	Pass criterion
1	`scripts/safe-precommit.sh run --all-files`	Exit 0, no `files were modified by this hook`
2	`./scripts/run-backend-tests.sh --summary`	Exit 0, 0 failures
3	`python3 scripts/audit-tests.py backend/tests` + `python3 scripts/audit-tests.py --selftest`; `cd frontend && npm run test:audit && npm run test:audit:selftest`	Exit 0, no `SELF-TEST BROKEN`, DEFERRED (backlog) count not increased vs the prior run
4	`./scripts/frontend-check.sh --no-claude --check-only`	Exit 0
5	`cd docs-site && npm run build`	Exit 0
6	`./scripts/validate-deployments.sh --json`	Every permutation `ok`, no "documented flag with no matrix entry"
7	`python3 scripts/release/check-version-consistency.py`	All version sources agree, single Alembic head
8	`backend/venv/bin/python3 scripts/audit-route-coverage.py --json`	Uncovered-route count not increased vs the prior run

This is exactly what ./scripts/release.sh preflight <v> and verify <v> already run as part of cutting a release. Run this stage standalone when you are not cutting a release; the release pipeline runs the equivalent automatically as part of its own gates.

Stage 2 — Dev-mode integration

~2.5-4h total across four stack cycles. Each cycle is a separate ./opentr.sh start dev ... invocation because the overlay combinations genuinely conflict (see 2B) or because isolating them keeps a failure attributable to one leg.

Cycle 2A — baseline + LLM + auth (one stack start, ~90-120 min)

./opentr.sh start dev --with-mock-llm --with-llm-test --with-ldap-test --with-keycloak-test --with-authentik-test

These five overlays safely co-run on one stack: each binds a distinct loopback port (mock-llm 5199, --with-llm-test's vLLM 5195, lldap 3890/17170, Keycloak 8180, Authentik 9022), none of them touches Celery worker scaling or COMPOSE_PROFILES (that's Cycle 2B's job), and only --with-llm-test takes a GPU — pinned via LLM_TEST_GPU_DEVICE_ID.

⚠️ LLM_TEST_VLLM_GPU_UTIL <= 0.45 alone does NOT make the default model fit a 12 GB card — it does not fit at ALL, at any tested setting (#608). Measured on an idle RTX 3080 Ti (11.62 GiB visible to the container, every other GPU-resident container stopped first so the entire card was free):

Loading the model's weights alone (--dtype float16 --quantization awq, both hardcoded in docker-compose.llm-test.yml) consumes ~8.85–9.3 GiB, independent of --gpu-memory-utilization — that flag only bounds the planned KV-cache pool, never weight loading.
With weights loaded (~9.32 GiB) and the default VLLM_COMPILE mode's inductor autotuning then running, the engine crashes trying to allocate another ~1.25 GiB of scratch (~10.87 GiB in use when the allocation is attempted) — a floor of roughly ~10.6–10.9 GiB against the 11.62 GiB visible, and this crash is not bounded by --max-num-batched-tokens, --max-num-seqs, or --max-model-len either.

So on a single-usable-12GB-GPU host, leg 4 (real vLLM) of this cycle is currently ⊘ NOT MEASURED — do not spend time chasing a GPU_UTIL value that will fit; none does. Two options, neither verified yet:

Set LLM_TEST_VLLM_EXTRA_ARGS=--enforce-eager (compose passthrough added alongside this doc fix) to disable torch.compile/CUDA graph capture — the standard vLLM VRAM workaround, and the escape hatch that did not exist when #608 was measured. This removes the ~1.25 GiB compile overhead but was not re-measured against a 12 GB card; confirm it actually fits before trusting it as the new guidance.
Point LLM_TEST_MODEL at a smaller model that is known to fit instead.

On a multi-GPU host (this project's documented reference hardware: GPU 1 for transcription/diarization, GPUs 0/2 reserved A6000s with 49 GiB each), point LLM_TEST_GPU_DEVICE_ID at one of the larger idle cards — the default model is proven there (compose file header) — or sequence the LLM legs (3-4 below) after the transcription-heavy legs (1-2) finish if only the transcription GPU is available.

Run these legs serially against that one stack. LLM provider and auth method are both single-valued DB-backed SystemSettings, so concurrent legs would race each other's config — each leg restores its own configuration on exit.

scripts/test-matrix.sh --only 2a runs legs 1, 2, 3 and 5 of this table as one command:

scripts/run-dev-tests.sh --full --all-overlays --search-quality && scripts/run-auth-e2e.sh --cleanup --skip-pki

run-dev-tests.sh is what supplies the overlay orchestration (it starts mock-llm / lldap / keycloak itself, reconciles the auth_config rows they need, and restores them on exit), which is why the leg goes through it rather than calling run-integration-tests.sh directly; --search-quality exists on that script specifically so this leg matches row 1 exactly. Leg 3's chat suites run inside row 2's run-e2e.sh with the mock-LLM overlay up. Row 4 (real vLLM) is deliberately NOT in that chain — --with-llm-test reserves a real GPU and is excluded from run-dev-tests.sh's overlay table by design, and on a single 12 GB card the default model does not fit at any tested setting (#608, above). Run it separately on a multi-GPU host, or with a confirmed-fitting LLM_TEST_MODEL.

#	Command	Pass criterion
1	`./scripts/run-integration-tests.sh --coverage --search-quality --cleanup`	Exit 0; the search-quality phase reports a non-zero collected-test count
2	`./scripts/e2e/run-e2e.sh`	0 failed. Visual-regression baseline failures must be explicitly triaged before release — never ignored as "known flaky"
3	`pytest backend/tests/e2e/test_chat.py test_chat_grounding.py test_chat_trace_panel.py` against `mock-gpt`/`mock-echo`/`mock-error`/`mock-reasoning`	Citations resolve, redaction masks apply, SSE completes, each error model surfaces the error it models
4	Same three files, provider repointed at `http://llm-test-vllm:8000/v1`	Real citations resolve to real segment ids; the local-provider redaction exemption fires (no masking of local-model input). ⊘ NOT MEASURED on a single 12 GB GPU — the default model does not fit at any tested setting (#608, see above); run it on a multi-GPU host or a confirmed-fitting `LLM_TEST_MODEL`
5	`./scripts/run-auth-e2e.sh --cleanup --skip-pki` (PKI is Stage 3 only — no dev-mode PKI variant exists)	Per-method summary green; `GET /api/auth/session` returns 200 anonymous afterward, proving config was restored
6	—	The `RUN_*`-gated security suites (both FIPS modes) already run inside leg 1's `run-integration-tests.sh` — do not re-run them separately here

Cycle 2B — GPU scaling (separate stack cycle, mandatory, ~30-45 min)

./opentr.sh stop
./opentr.sh start dev --gpu-scale

This is a genuine conflict with 2A, not a convenience separation: --gpu-scale sets COMPOSE_PROFILES=gpu-scale and swaps in docker-compose.gpu-scale.yml's worker topology (N parallel Celery workers pinned to GPU_SCALE_DEVICE_ID), which is additive with none of 2A's overlays. On a single-GPU host, set GPU_SCALE_DEVICE_ID to that GPU and stop --with-llm-test's vLLM first — both want VRAM.

Run via scripts/gpu-scale-smoke.sh. docker-compose.gpu-scale.yml runs exactly ONE celery process (gpu-scaled@%h) with --concurrency=GPU_SCALE_WORKERS, so "N workers register in Flower" was never the right pass criterion — it checks the pool's max-concurrency on that one process instead, plus the optional default worker (gpu-transcription@%h) when GPU_SCALE_DEFAULT_WORKER=1. Pass: the gpu-scaled@* worker is present in Flower's /api/workers?refresh=1 (⚠️ not the unrefreshed /api/workers — that endpoint is a one-shot snapshot cached at Flower's own process startup and never re-inspects on a timer, so a worker still importing torch/whisperx when Flower booted is absent from it forever; see #609) with stats.pool.max-concurrency == GPU_SCALE_WORKERS, at least 3 concurrent uploads all reach completed, no CUDA OOM string in celery-worker-gpu-scaled logs during the run, and batch wall-clock is less than N times a single-file baseline.

diar-native routing under gpu-scale / gpu-split — HARDWARE-GATED, last verified 2026-09-05

These two rows cannot be cleared by docker compose config or by "a job completed": the PyAnnote fallback is silent by design, so an unreachable sidecar still yields correct transcripts (#655, #711). Static wiring is enforced by backend/tests/unit/test_diar_native_overlay_wiring.py; the live proof is backend/tests/integration/test_diar_native_multigpu_provider_live.py, which is integration/gpu-marked and therefore never runs in CI — it must be run by hand on multi-GPU hardware:

cd backend && POSTGRES_PORT=<offset+5176> BACKEND_PORT=<offset+5174> \
  venv/bin/python -m pytest tests/integration/test_diar_native_multigpu_provider_live.py \
  --override-ini="addopts=" -m "integration or gpu" -v

⚠️ Bring up exactly one diarizing topology per run. With celery-worker-gpu-scaled and celery-worker-gpu-diarize both up, the broker picks the worker and the result is unattributable; the probe skips as NOT MEASURED rather than passing.

Row	Last verified	Evidence
`--gpu-scale` reaches diar-native	2026-09-05	`celery-worker-gpu-scaled`: `native diarization done in 2.8s`, `diarization_provider == "native"`, 0 fallback lines
`--with-gpu-split` reaches diar-native	2026-09-05	`celery-worker-gpu-diarize`: `native diarization done in 3.3s`, 0 fallback lines; `celery-worker-gpu-transcribe` diarized nothing, i.e. it forwarded

Criterion 5 of #711 — VERIFIED on real hardware, 2026-09-05 (cross-card). The device relationship between the sidecar and the gpu-scale worker was the one remaining unmet criterion: GPU_SCALE_DEVICE_ID defaults to 2 while the sidecar defaults to DIAR_NATIVE_GPU:-${GPU_DEVICE_ID:-0}, so the shipped defaults describe a cross-card arrangement. The earlier 2026-09-05 run above had both on one card (GPU 1) because the host was believed to have only one usable GPU for this project; that was out of date — a second RTX A6000 (GPU 2) is idle and available (GPU 0 stays off limits: tritonserver + an unrelated container).

opentr.sh gained a --diar-native-gpu N flag for this (applied after --gpu-device, so the two can differ — --gpu-device alone pins every GPU_DEVICE_VARS entry, including DIAR_NATIVE_GPU, to the same value):

./opentr.sh start dev --fresh xcard711 --gpu-scale --port-offset 300 \
  --gpu-device 2 --diar-native-gpu 1
# resolves to: GPU_SCALE_DEVICE_ID=2 (RTX A6000), DIAR_NATIVE_GPU=1 (RTX 3080 Ti)

cd backend && POSTGRES_PORT=5476 BACKEND_PORT=5474 \
  venv/bin/python -m pytest tests/integration/test_diar_native_cross_card_placement_live.py \
  --override-ini="addopts=" -m "integration or gpu" -v

Ground truth was taken by PID, not by the device_ids: reservation alone: docker top otfresh-xcard711-diar-native (pid 3559215) and docker top otfresh-xcard711-celery-worker-gpu-scaled (pid 3558939) were cross-referenced against nvidia-smi --query-compute-apps=pid,gpu_uuid + nvidia-smi --query-gpu=index,uuid — the sidecar's pid held a CUDA context on GPU-272aeadc… (nvidia-smi index 1, RTX 3080 Ti, 808 MiB) and the worker's pid on GPU-5933f022… (index 2, RTX A6000, 1318 MiB): two different physical cards, confirmed. Two diarizations in that arrangement both went through otfresh-xcard711-celery-worker-gpu-scaled, logged native diarization done in 2.6s / 1.5s (a third run afterward: 0.7s / 0.6s), diarization_provider == "native" on both rows, and zero falling back to pyannote lines.

Finding: cross-card WORKS and is not measurably slower. All four cross-card runs (0.6–2.6s) overlap the same-card baseline above (2.8s) — the loopback hop between two containers on the compose bridge network costs nothing next to GPU compute time on this host. This is one measurement on one host under light load, not a general throughput claim under contention. New live probe: test_diar_native_cross_card_placement_live.py (integration/gpu-marked, same CI exemption as its sibling); unmarked companion guard: test_opentr_diar_native_gpu_override.py.

The default worker (celery-worker, scaled up via GPU_SCALE_DEFAULT_WORKER=1) is ALSO diarize-capable and was up alongside celery-worker-gpu-scaled in this environment, which reproduces the same broker-picks-the-worker ambiguity #764 already documented for gpu-scale vs. gpu-split — it was stopped with docker stop otfresh-xcard711-celery-worker (not signaled, not reconfigured) before the measurement above so the result stayed attributable.

Cycle 2C — diarization providers (~20 min, can fold into 2A if VRAM allows)

diar-native loads by default — --no-diar-native is what suppresses it — so Cycle 2A already exercises it; this is not a separate opt-in overlay. Add:

./scripts/diar-native-smoke.sh

Pass: the diar-native container holds non-zero device memory and shows no restart loop. Then run one transcription with --no-diar-native to prove the PyAnnote fallback still works. Pass: completes, speakers assigned.

Cycle 2D — lite / CPU-only (~20 min)

./opentr.sh stop
./opentr.sh start dev --lite --fresh litecheck --port-offset 200

Run via scripts/lite-smoke.sh. Pass: stack healthy, no celery-worker-gpu* container in docker ps, no stack process holding memory on any GPU (nvidia-smi). A transcription pass needs a cloud ASR key — if one isn't configured, record ⊘ NOT MEASURED, never a pass. Cover --cpu mode with the same script and the same criteria.

⚠️ This cycle is TOPOLOGY-only — it does not exercise the pipeline. lite-smoke.sh proves the absence of a GPU worker and of resident GPU memory, and needs a real cloud ASR key to go further than that. It does not upload a file, run ASR, index it, search it, or chat over it. The actual upload -> ASR -> segments/speakers -> search -> chat pipeline for a lite deployment is covered separately, with no vendor key required, by the "3-lite" leg below (scripts/release-tests/test-lite-mode.sh) — see that section for what it asserts.

Stage 3 — Deployment mode (prod)

~3-5h. Requires the dev stack STOPPED.

./opentr.sh stop
BUILD_MODE=local PUSH_LATEST=false ./scripts/docker-build-push.sh all
./scripts/release-tests/test-fresh-install.sh --yes
REQUIRE_PREVIOUS=1 ./scripts/release-tests/test-upgrade.sh --yes

Pass: every assertion in each scenario's REPORT.md is PASS. This sequence is exactly what ./scripts/release.sh rehearse <v> runs — that is the preferred invocation, since it owns the ledger and records the run against a real version. Use the raw commands above only when rehearsing outside a release cut.

scripts/test-matrix.sh --only 3 --yes runs the same sequence by calling the stage script both callers share:

scripts/release/65-rehearse.sh "$(tr -d '[:space:]' < VERSION)"

That is one engine with two callers, not two implementations: 65-rehearse.sh runs Scenario A, tears A's stack down so B can bind the stock ports, then runs Scenario B. It is also pure — only release.sh writes the .release/<version>/ ledger — so running it from the matrix cannot corrupt a real release's recorded state. It replaced a leg whose description said "fresh-install + upgrade" while its command was test-fresh-install.sh alone.

Stage 3 — lite-mode full rehearsal

~30-45 min. Requires the dev stack STOPPED (same one-liner-defaults constraint as test-fresh-install.sh/test-upgrade.sh — see scripts/release-tests/README.md).

./opentr.sh stop
./scripts/release-tests/test-lite-mode.sh --yes

Runs the real docker-compose.lite.yml (no-GPU, cloud-ASR-only) topology against a mocked cloud ASR provider (scripts/mock-asr-server.py, a Gladia stand-in) and a mocked LLM (scripts/mock-llm-server.py), so it needs no GPU, vendor API key, or network egress. Where Cycle 2D's lite-smoke.sh only proves the no-GPU topology, this leg drives the real pipeline: ASR config creation, file upload, transcription completion against the canned mock transcript, segment count / speaker count / distinctive-token assertions, hybrid search, an OpenSearch ML deployed-model check, a chat turn against the mocked LLM with a real citation, the Alembic head, and a negative-path (MOCK_ASR_SCENARIO=error) upload reaching error status with no leaked credential in the error message.

Pass: every assertion in REPORT.md is PASS. Known gap, deliberately deferred (see backend/tests/CLAUDE.md): the mock's per-request ?scenario= override is not reachable from GladiaProvider itself (issue tracked in the "known deviation" note in backend/tests/integration/test_lite_mode_mocked_providers.py), so this leg's negative-path check restarts the mock-asr container with MOCK_ASR_SCENARIO=error mid-run rather than driving it per-request — safe here because, unlike the shared pytest module, this script owns its own compose lifecycle serially.

PKI/mTLS is prod+nginx ONLY. There is no dev-mode variant and none should be invented — Vite cannot terminate mTLS.

./scripts/pki/setup-test-pki.sh
./opentr.sh start prod --build --with-pki
RUN_PKI_E2E=true pytest backend/tests/e2e/test_pki.py -v

Pass: a client cert from scripts/pki/test-certs/clients/*.p12 authenticates at https://localhost:5182; a request with no cert is rejected at the TLS layer (nginx), not served as an anonymous 200.

scripts/test-matrix.sh --only 3-pki --yes runs scripts/pki/run-pki-e2e-leg.sh, which owns those three commands plus the orchestration they need and nothing else: it clears any release-test stack still holding the stock opentranscribe-* names from an earlier Stage 3 leg (via each scenario's own labelled --cleanup, which cannot reach production volumes), generates the CA, brings up prod+nginx+PKI, waits for the mTLS listener on 5182, runs the suite with RUN_PKI_E2E=true (without which test_pki.py skips wholesale and the leg would "pass" having run nothing), and stops the stack again. --keep-stack leaves it up for inspection. It follows the standard exit contract: 0/1/2/3/4.

Scope decisions, stated explicitly:

No separate prod pass is run for --lite/--gpu-scale. Compose validity is checked in Stage 1.6, runtime behavior in 2B/2D, and prod images behave identically to dev images for those flags. This is a deliberate scope decision, not an oversight.
Offline/air-gapped deployment is config-validated only (Stage 1.6). A real network-namespaced offline install pass is a known, currently uncovered gap — do not read Stage 1.6 as proving offline mode works end to end.
opentranscribe.sh backup/restore and opentranscribe.sh update --rollback ARE rehearsed, by test-upgrade.sh's phases 13-17 (#598). Phase 12 asserts the rollback precondition (# OT_PREVIOUS_IMAGE_TAG) a real update --version records; phase 15 restores the phase-06b pre-upgrade backup over damage inflicted through the real API and asserts content digests, not row counts, match exactly (a delete+insert pair leaves counts unchanged); phase 16 runs the real update --rollback and asserts the FROM image serves the restored FROM database through its real API (login, file list, transcript text) — not merely that the command exited 0; phase 17 proves the documented recovery loop (roll back -> re-upgrade) completes cleanly. A ROLLBACK_INJECT_FAULT self-check (truncate/no-damage/stale-oracle) deliberately breaks the tail so its own failure detection is exercised for real — see scripts/release-tests/selftest-rollback-fault-injection.sh. Still NOT covered, deliberately: backup --encrypt (unattended gpg needs a passphrase file the CLI does not support), the in-app scheduled-backup system (app/services/backup_service.py — a separate implementation with its own real unit/API coverage but no end-to-end restore proof), and MinIO/OpenSearch restore (the DB restore does not touch either — asserted as R-11, not merely unclaimed). Full detail: scripts/release-tests/README.md.

Test data & fixtures

What each stage actually uploads/transcribes, for developers extending this matrix or writing a new rehearsal leg — none of it needs network access or a committed binary asset beyond what's already in the repo:

Stage / leg	Data source	Why
Stage 1, Cycle 2C	none (config/compose validation, or GPU memory check only)	no media needed
Cycle 2A (e2e, chat)	`backend/tests/e2e/conftest.py`'s `owned_media_factory` — uploads an ephemeral file per test, deleted in its own teardown	e2e tests must never persist changes to dev data (root `CLAUDE.md`)
Cycle 2A legs 3-4 (chat)	canned scenario responses from `scripts/mock-llm-server.py` (`mock-gpt`/`mock-echo`/`mock-error`/`mock-reasoning`), or a real `llm-test-vllm` model for leg 4	exercises the app's actual error handling, not a stub that always succeeds
Cycle 2D	none — `lite-smoke.sh` is topology-only (no GPU worker, no resident GPU memory); it does not upload anything	see the "TOPOLOGY-only" warning above
Stage 3 fresh-install / upgrade	`scripts/release-tests/provision-test-media.sh` — derives small (under 5 MB) real-speech clips from real-speech assets already in the repo via `ffmpeg`, never a network fetch. Rejects the e2e suite's synthetic 440 Hz sine and `watch/podcast.mp3` (both silent-of-speech) specifically because the scenario's core assertion is "transcript is non-empty," which a tone or synthetic clip can't honestly prove	reproducible on any checkout, no decaying external URL, and the assertion means something because the source audio genuinely has speech
Stage 3 lite-mode	`scripts/mock-asr-server.py`'s canned Gladia-shaped response, reshaped from `backend/tests/fixtures/media/sample_transcript.json`, containing the distinctive token "Zylofenix" so search/citation assertions can confirm the right content actually round-tripped	no GPU, vendor API key, or network egress required for the whole `--lite` pipeline rehearsal
Stage 3 PKI	`scripts/pki/test-certs/clients/*.p12`, generated by `scripts/pki/setup-test-pki.sh` (gitignored, contains private keys — never commit)	real client-cert auth, not a stub

If a future leg's fixture needs are not covered by one of the above (e.g., a rehearsal that needs a specific language, a specific speaker count, or a specific duration), extend provision-test-media.sh's SOURCES list or add a new derivation step there rather than committing a new binary fixture or reaching for a network URL — that script's header explains the reasoning in full.

Stage 4 — Image/release gates

~45-90 min. Mostly already automated — this stage is "confirm existing automation covers it," not new work.

./scripts/release.sh scan <v>
./scripts/release.sh build <v>       # multi-arch; needs USE_REMOTE_BUILDER=true, else 2-3h under QEMU
./scripts/release.sh publish <v>
./scripts/release.sh promote <v>

scan's security-tooling check is a warn-severity preflight, not a blocking one — a missing trivy, grype, or syft on PATH silently reduces scan coverage rather than failing the stage. Verify all three are installed before starting Stage 4.

scripts/test-matrix.sh --only 4 runs the scan for real:

scripts/release/50-scan.sh "$(tr -d '[:space:]' < VERSION)"

The stage script, not release.sh scan, for the same reason Stage 3 calls 65-rehearse.sh: 50-scan.sh does the work, release.sh owns the .release/<version>/ ledger, so the matrix can run the gate without writing a ledger entry a real release would then read as current. The dispatcher warns up front about any of the three scanners missing from PATH, so a thin scan does not read as a thorough one. build/publish/promote remain release.sh-only: they reach Docker Hub and GitHub, and nothing outside the release pipeline should invoke them.

Time budget

Stage	Time
Stage 1	6-9 min
Cycle 2A	90-120 min
Cycle 2B	30-45 min
Cycle 2C	20 min
Cycle 2D	20 min
Stage 3 (fresh-install + upgrade)	3-5 h
Stage 3 (lite-mode rehearsal)	30-45 min
Stage 4	45-90 min (2-3 h if the remote builder is unavailable)
Full matrix	~7-9 h — realistically one working day with triage

Sequencing: Stage 1 + Cycle 2A fit in one day. Cycles 2B/2C/2D plus Stage 3 fit in a second day. Stage 4 happens naturally as part of an actual release.sh run — don't schedule it separately.

Anti-staleness

scripts/test-matrix.sh parses this document's leg tables and fails loudly if a documented leg has no matching implementation in the script, or vice versa — the same technique scripts/validate-deployments.sh uses to keep its deployment matrix from drifting out of sync with opentr.sh. Don't add a row here without adding its leg to the dispatcher in the same change, and don't add a leg to the dispatcher without a row here.

CI/CD readiness

Every leg is non-interactive (--yes bypasses confirmation prompts).
Exit codes follow release.sh's stable contract: 0 pass, 1 gate failed, 2 misuse, 3 precondition unmet, 4 operator abort.
--json output matches release.sh's shape: {stage, leg, status, criteria[], next[]}.
State is held in a gitignored per-run ledger; nothing persists across invocations except that.
Stage 1 needs no GPU and is the CI-safe subset. Stages 2-4 need a GPU runner and a stack that only one job touches at a time — serialize, never fan out in parallel.
No AWS-specific logic, credentials, registry names, or pipeline definitions belong in this repo. The core is vendor-clean (see backend/CLAUDE.md's note on the absence of app/services/cloud). A future consumer — the private opentranscribe-cloud repo, or a CI job — shells out to scripts/test-matrix.sh <stage> --json --yes and reads the exit code. That's the entire contract surface this repo owns, in the same spirit scripts/release/release-criteria.yaml states for its own gates: "If a second consumer ever appears (a CI job, an AWS promotion job), add its stage here and wire it the same way — bidirectionally, or not at all."

scripts/test-matrix.sh RUNS every leg​

Coverage stance at a glance​

Stage 1 — Static / no live stack​

Stage 2 — Dev-mode integration​

Cycle 2A — baseline + LLM + auth (one stack start, ~90-120 min)​

Cycle 2B — GPU scaling (separate stack cycle, mandatory, ~30-45 min)​

diar-native routing under gpu-scale / gpu-split — HARDWARE-GATED, last verified 2026-09-05​

Cycle 2C — diarization providers (~20 min, can fold into 2A if VRAM allows)​

Cycle 2D — lite / CPU-only (~20 min)​

Stage 3 — Deployment mode (prod)​

Stage 3 — lite-mode full rehearsal​

Test data & fixtures​

Stage 4 — Image/release gates​

Time budget​

Anti-staleness​

CI/CD readiness​