testing.md

Codexa testing

Local pytest invocations, the bench opt-in marker, and the GitHub Actions / SonarCloud wiring.

← README · Decision records: ADR 0004 (why the suite is offline-only), ADR 0005 (the seam modules tests patch), ADR 0010 (the reset entry point fixtures drain)

Table of contents

1. Testing
2. Continuous integration

Testing

Run the Python pre-finish gate in this order from the repository root. Pyright reads the project configuration, so do not pass positional source paths. Select only tests that cover the changed behavior. Full, broad-coverage, benchmark, and long suites run only when explicitly requested. Never start separate pytest commands concurrently; pytest-xdist owns safe parallelism inside one invocation.

uv run --frozen ruff check codexa tests
uv run --frozen pyright
uv run --frozen python scripts/check_repo_privacy.py
uv run --frozen pytest tests/test_changed_contract.py

Pytest uses four xdist workers and module-scoped scheduling by default. Pass -n 0 for a focused test that owns process-global state. Mutation testing does this automatically. Explicit opt-in suites are:

HYPOTHESIS_PROFILE=deep uv run --frozen pytest $(ls tests/*fuzz*.py tests/security/test_*.py)
uv run --frozen pytest -m bench
uv run --frozen pytest -m long -n 0
uv run --frozen pytest --cov=codexa --cov-report=term --cov-fail-under=90

The default suite is offline-only by design (mirroring the product's no-default-egress invariant, ADR 0004): CODEXA_DISABLE_HF=1 and TOKENIZERS_PARALLELISM=false are set in tests/conftest.py, NLTK calls in metadata.entities are mocked, and SentenceTransformer / Chroma / Ollama / paddleocr / tesseract are all replaced with deterministic stand-ins.

Hypothesis property tests run inside the normal suite with the ci profile (deadline=None, cheap example budget). Suite-size figures are not hard-coded because parametrization and generated cases make them drift. Recompute the current inventory from the repository root when a release or audit needs it:

rg -n '^def test_|^[[:space:]]+def test_' tests | wc -l  # definitions
rg -n '@given\(' tests | wc -l                            # property sites
uv run --frozen pytest --collect-only -q                  # collected cases

When publishing a count, record the command and commit hash that produced it. For adversarial sweeps, run HYPOTHESIS_PROFILE=deep uv run --frozen pytest ...; the scheduled/manual .github/workflows/deep-fuzz.yml lane applies that profile without slowing every pull request. That lane discovers its targets at run time — every tests/**/*fuzz*.py plus tests/security/test_*.py — so a new fuzz file is picked up automatically (no YAML edit needed). The ls glob above mirrors that discovery for a local run.

The bench marker is deselected by default. Tests under it spawn the real indexer subprocess against BENCH_CORPUS_DIR (default /path/to/corpus) and measure throughput, system pressure, and persisted store bytes per chunk; they are the test-runner counterpart of scripts/bench_indexer.py. See tests/test_bench_indexer.py for the full env-var override list.

indexer-benchmark.yml runs weekly and on manual dispatch, never on pull requests. It builds a deterministic 120-file Markdown/text/HTML corpus, exercises the real extraction → MiniLM embedding → Chroma store path, and fails when throughput drops below 0.03 files/s or persisted growth exceeds 1,000,000 bytes/chunk. The indexer-benchmark-evidence artifact retains the JSON summary and structured indexer logs for 30 days. Adjust a threshold only from measured runner evidence, and keep the prior artifact with the change rationale.

⚠️ Do not run two separate pytest commands concurrently against this repo. Each pulls in chromadb + pymupdf + sentence-transformers. Use the configured xdist workers inside one command; force -n 0 for stateful tests.

Mutation testing changed production lines

Line coverage only proves that code ran. For each production-logic change, mutate only the executable lines changed in that work item and use the smallest focused tests that exercise them:

uv run --frozen python scripts/run_mutation.py --jobs 1 --lines 58-69 codexa/utils/lru.py tests/test_mutation_lru_2026_08_03.py

--lines is a changed-line slice and deliberately skips the whole-module ledger. The slice must report zero survivors; document any genuinely equivalent mutant in the focused test instead of widening the test set to unrelated suites. A timeout is an inconclusive failure, never evidence that a test killed the mutation: every mode reports it separately and exits nonzero. --check does not compare an inconclusive run with the ledger, --update-baseline does not write it, and --check-all continues the remaining modules before failing the overall gate.

For a module already recorded in scripts/mutation_baseline.json, the explicit whole-module non-regression check is:

uv run --frozen python scripts/run_mutation.py --jobs 1 --check codexa/utils/lru.py tests/test_mutation_lru_2026_08_03.py

--check is a survivor-count ratchet: it fails when a recorded module exceeds its allowed survivors. An unrecorded module passes with an invitation to establish a reviewed baseline, so this mode is not a universal zero-survivor claim. A bare whole-module run remains the authoring mode and fails on any survivor.

Use --update-baseline only in a separately scoped whole-module mutation audit to establish a reviewed first baseline after every survivor is killed or documented as equivalent, or to lock in a genuine decrease for an existing entry:

uv run --frozen python scripts/run_mutation.py --jobs 1 --update-baseline codexa/utils/lru.py tests/test_mutation_lru_2026_08_03.py

You must never raise a baseline to make a run pass. Do not create a baseline merely because a module is new. The scheduled/manual mutation workflow runs --check-all --jobs 4 in its controlled CI lane; it is not part of the per-push or local pre-finish gate.

Conventions

• Phase regression tests — every shipped phase lands one regression file under tests/test_phase{N}_{slug}_{YYYY_MM_DD}.py covering the closed TODO row(s). The file's docstring lists the closed IDs + the contract pinned. Phase 85+ examples: ROB-* atomic writes, WIRE-18/19/20/21/23/24/25, PROMPT-5..10, FAITH-5/7+COH-8, PERF-30/31/33/34/35, FAITH-8/9, RETQ-7/8/9/10, COH-9..12+BOT-28/29/30, CACHE-2/3/8/9/12+SCAL-15/16.
• AppContext.reset between tests — when a test mutates a wired cache, drain the chain via AppContext.current().reset() (or the per-cache clear_<name>_cache() helper) in a fixture teardown so subsequent tests start cold (ADR 0010).
• Type ignores on duck-typed test stubs — SimpleNamespace / Mock objects swapped in for Session / Turn / BM25Searcher etc. carry a # type: ignore[arg-type] (or similar) marker; pyright's stub-vs-real-type mismatch is intentional.

Continuous integration

GitHub Actions runs on every push and pull request to develop / main. The workflow has seven jobs:

• lock — uv lock --check, so a pyproject.toml edit that was never re-locked fails before anything is installed and every later job resolves from the committed uv.lock.
• artifacts — builds twice from clean git archive source trees, compares byte-identical wheels and metadata-normalized sdists, rebuilds each wheel from its sdist, and uploads the first pair. It installs the wheel into a fresh venv from locked runtime dependencies, checks every packaged catalog/data/UI resource, and smokes codexa --help plus codexa demo --no-write outside the checkout. The wheel itself drives dependency resolution; the exported lock constrains every resolved runtime dependency and supplies its hashes.
• optional-install — uv sync --frozen --all-extras on Python 3.12, after installing declared native prerequisites, so an unsupported optional package cannot silently break the supported install profile.
• lint — ruff check codexa tests on Python 3.12.
• typecheck — uv run --frozen pyright on Python 3.12. This is a hard gate; a type regression blocks merge.
• test — one ordered Python 3.12 job. It first runs pytest over the default non-long suite with xdist, --cov-fail-under=90, and XML coverage. It then runs the deterministic repo-doc retrieval harness smoke: codexa eval docs-baseline --save … --min-recall 0.5 --min-ndcg-at-5 0.3 --max-regression 0.02. That model-free check covers bundled-case loading, metric aggregation, fixed floors, and saved-summary diffing; it does not claim to measure production ranking. Last, CI stages seven bundled documents (three repository-root files and four files under docs/) into the corpus configured by .codexa-quality/ci-retrieval.yml, builds a real MiniLM/Chroma index, and runs codexa eval baseline through dense retrieval, persisted corpus BM25, RRF, and MMR. Its fixed floors and --max-regression 0.02 ratchet compare against .codexa-quality/real_pipeline_eval_baseline_summary.json, uploaded as production-retrieval-baseline. This seven-document gate measures production-path retrieval quality, not 200k-file scale. The main pytest run separately includes synthetic 100,000- and 200,000-file RSS gates that remain fast enough for the default lane: test_scal22_memory_budget_2026_07_20.py covers the scan/change handoff, and test_mem21_memory_budget_2026_07_20.py covers fresh and unchanged current-run ledgers. Those tests generate paths and metadata without indexing 200,000 real documents. The bundled relevance lists are explicitly open (closed_gold: false), so precision is neither published nor gated. Set closed_gold: true and add --min-precision for an operator corpus only when expected_sources exhaustively judges that corpus; omitted closed_gold defaults to true for legacy files. Answer-generating evals should also label each row answerable: true or answerable: false; omitted labels stay unknown and do not enter refusal gates. Use --max-false-refusal-rate for answerable cases and --min-refusal-recall for unanswerable cases. Raw refusal rate remains diagnostic only because refusing every question is not success.
• sonar — SonarQube Cloud analysis (needs test), ingesting the coverage XML. Skipped on fork PRs (no SONAR_TOKEN).

codexa eval tune-hybrid --cases … is not a CI job — it is an operator-run command. It selects the fusion grid winner by mean NDCG@5, using mean MRR@5 as the tie-breaker; the cutoff stays fixed while rrf_k is swept. Open and closed qrels both contribute ranking metrics; only closed qrels contribute the retained precision diagnostic.

Two scheduled/manual workflows stay outside the per-push gate: deep-fuzz.yml runs the Hypothesis Deep Fuzz job, and indexer-benchmark.yml runs the representative production-path index benchmark.

Concurrency is set so a new push to a branch cancels any in-flight run, and the same OMP_NUM_THREADS/MKL_NUM_THREADS/TOKENIZERS_PARALLELISM discipline used locally is applied as workflow-level env vars to keep CI minutes predictable. See .github/workflows/ci.yml.

sonar-project.properties configures SonarQube / SonarCloud: production code is codexa/; tests/ and examples/ are declared as test sources but excluded from the main quality analysis so test idioms (long parametrize lists, repeated fixture scaffolding) don't surface as code smells. The coverage XML emitted by the CI pytest run feeds Sonar via sonar.python.coverage.reportPaths.