Plan: Wave-pattern test harness foundation

[bakeb7j0]

Plan: Wave-pattern test harness foundation

Goal

Ship a runnable nightly + on-demand test harness for the wave-pattern pipeline that exercises Tier 1 (unit) and Tier 4 (e2e campaign) coverage end-to-end across GitHub and GitLab fixtures, closing the campaign-as-test-of-last-resort gap surfaced by Plan #607 (Beta).

Scope

In scope

• New repo Wave-Engineering/ccwork-testtarget with Python + pytest runner, fine-grained PAT auth, teardown helpers.
• Tier 0 static checks: required CLIs on PATH, MCP tools/list snapshots, skill frontmatter parse, MEMORY.md integrity, WAVE_AXIOMS.md presence and structure.
• Tier 1 unit wiring: existing bun test suite for each of four MCP servers (mcp-server-sdlc, mcp-server-discord, mcp-server-nerf, mcp-server-wtf); bus-script unit tests in tmpdir for wave-init, flight-finalize, changelog-aggregate, wave-cleanup; status-panel snapshot test.
• Fixture-repo lifecycle: GitHub fixtures private under Wave-Engineering/; GitLab fixture under gitlab.com/testtarget/. Per-run-id-prefixed create + prefix-scoped teardown.
• Tier 4 v0 end-to-end tests: 4.1 (single-flight, single-issue, GitHub); 4.8 (single-flight, single-issue, GitLab — parametrized 4.1 + platform assertions); 4.6 (cross-repo, GitHub → GitHub).
• Telemetry replay tooling — given a failed Tier 4 run, reconstruct timeline from ~/.claude/logs/mcp.jsonl + bus state into a single forensic doc.
• Runner modes: --keep-state (preserve bus dirs, worktrees, partial logs on failure) and --bisect <failure-tag> (re-run only the suspect step).
• Nightly cron + Discord report posting to a dedicated #harness-test channel.

Out of scope

• Tiers 2 / 3 unit-grain coverage — backfill on bisection pain in a follow-up Plan.
• Tier 5 chaos / fault-injection — separate follow-up Plan.
• Tier 4 tests 4.2–4.5 / 4.7 (multi-flight parallel; multi-flight with file conflict; multi-wave dependency chain; multi-phase Plan; KAHUNA on non-main base ref) — extend in follow-up Plan once v0 runner shape proves out.
• Epic Epic: Close the wave-pattern automated-testing gap (integration harness, daemon coverage, sub-agent contract enforcement) #626 sub-issues feat: complete repo scaffolding — CI, settings, sync, uninstall, changelog #2–docs: add foundation story checklist to PRD template #9 (daemon-process coverage, canonical-line enforcement, bus-path-scheme assertion, substring → exact-line audit, CHANGELOG-fragment guard, harness-backgrounding instrumentation, kahuna-parity preflight, auto-merge retry policy) — filed as standalone Stories directly under Epic: Close the wave-pattern automated-testing gap (integration harness, daemon coverage, sub-agent contract enforcement) #626 via /issue feature --epic 626, not part of this Plan.
• Prompt regression, cost regression, real-Discord-channel state, behavioral effectiveness of WAVE_AXIOMS — model-fidelity / observational concerns, not deterministic-CI surfaces.

Plan-level Definition of Done

• Phase 1 DoD satisfied
• Phase 2 DoD satisfied
• Nightly cron runs unattended on a dedicated host, posts a report to #harness-test
• All v0 tier tests pass green on a freshly-installed cc-workflow setup (proves the harness is portable across machines, not pinned to one operator's env)
• Fixture-repo lifecycle is per-run idempotent — create and tear down without state leak across nights
• Forensic-replay tooling produces a single doc on a failed Tier 4 run
• At least one open Plan Plan: Beta — Wave-pattern reliability + autonomy hardening (operational backlog campaign B) #607 follow-up bug (drawn from cc#629–bug(wave-pattern): GitLab wave::N labels and phases-waves.json issue lists can disagree — pick one as authoritative or surface drift #638, sdlc#425) has a corresponding red Tier 4 test that turns green when the bug fix lands — the harness's load-bearing self-validation contract

Phases

Phase 1 — Tier 1 runtime

DoD:

• Repo Wave-Engineering/ccwork-testtarget created (private), pytest skeleton in place, fine-grained PAT scoped to harness-prefixed names only
• Tier 0 static checks run and pass: CLIs on PATH, MCP tools/list snapshot match, skill frontmatter parse, MEMORY.md integrity, WAVE_AXIOMS.md presence
• Tier 1 wiring runs all four MCP server bun test suites and collects results
• Bus scripts have unit tests in tmpdir (per lesson_destructive_test_homedir.md)
• Status-panel snapshot test wired (regression for cc#631-class bugs)
• Nightly cron + Discord posting to #harness-test works
• Repo's own CI green on a freshly-cloned setup

Stories:

• Story 1.1: Foundation — repo bootstrap + Python package skeleton (Wave-Engineering/ccwork-testtarget#1)
• Story 1.2: MCPClient subprocess wrapper + JSON-RPC framing (Wave-Engineering/ccwork-testtarget#2)
• Story 1.3: NotificationSubsystem + @-mention guardrail (Wave-Engineering/ccwork-testtarget#3)
• Story 1.4: Structured event emission to mcp.jsonl (Wave-Engineering/ccwork-testtarget#4)
• Story 1.5: Tier 0 static checks (Wave-Engineering/ccwork-testtarget#5)
• Story 1.6: Tier 1 — MCP server bun test wiring (Wave-Engineering/ccwork-testtarget#6)
• Story 1.7: Tier 1 — bus-script unit tests in tmpdir (Wave-Engineering/ccwork-testtarget#7)
• Story 1.8: Tier 1 — status-panel snapshot test (Wave-Engineering/ccwork-testtarget#8)
• Story 1.9: Tier 6 observability checks (Wave-Engineering/ccwork-testtarget#9)
• Story 1.10: Tier-execution runner + --tier CLI (Wave-Engineering/ccwork-testtarget#10)
• Story 1.11: GitHub Actions CI for harness helper-code unit tests (Wave-Engineering/ccwork-testtarget#11)
• Story 1.12: Nightly cron deployment + state directories (Wave-Engineering/ccwork-testtarget#12)

Phase 2 — Tier 4 v0

DoD:

• Fixture-repo lifecycle works for both GitHub and GitLab: per-run-id-prefixed create + prefix-scoped teardown
• Test 4.1 (GitHub single-flight) passes end-to-end: kahuna branch created via wave_init, kahuna→main MR opened by wave_finalize, gate runs all 4 trust signals in a single tool-use block, pr_merge lands the kahuna→main MR, observability events captured in ~/.claude/logs/mcp.jsonl, status panel reflects terminal state, teardown clean
• Test 4.8 (GitLab single-flight) passes: parametrized 4.1 + GitLab-specific assertions — glab adapter parity for pr_create / pr_merge / pr_status / pr_diff / pr_files / pr_wait_ci / pr_merge_wait; merge-train-warning emitted to Discord before pr_merge; approval rule scoped via protected_branch_ids permits the auto-merge; skip_train: true interpreted per platform (silent passthrough on GitLab; bypass-queue on GitHub)
• Test 4.6 (cross-repo GitHub → GitHub) passes: pre-created worktrees in target repo, gh -R scoping on every command, no isolation: "worktree" flag misuse, kahuna branches in BOTH repos, both kahuna→main MRs land, wave-status state lives in master plan repo, worktree teardown unlocks before force-removal
• Telemetry replay tooling produces a single forensic doc on a deliberately-broken run
• --keep-state flag preserves bus dirs, worktrees, partial logs on failure
• --bisect <failure-tag> mode re-runs only the suspect step

Stories:

• Story 2.1: FixtureLifecycle helper (GitHub per-run-id, GitLab long-lived) (Wave-Engineering/ccwork-testtarget#13)
• Story 2.2: ForensicGenerator + KeepState helpers (Wave-Engineering/ccwork-testtarget#14)
• Story 2.3: CostTracker + pricing.yml + budget overage detection (Wave-Engineering/ccwork-testtarget#15)
• Story 2.4: Tier 4 driver — claude CLI subprocess invocation (Wave-Engineering/ccwork-testtarget#16)
• Story 2.5: Tier 4 Test 4.1 — GitHub single-flight (Wave-Engineering/ccwork-testtarget#17)
• Story 2.6: Tier 4 Test 4.8 — GitLab single-flight (Wave-Engineering/ccwork-testtarget#18)
• Story 2.7: Tier 4 Test 4.6 — cross-repo (GitHub → GitHub) (Wave-Engineering/ccwork-testtarget#19)
• Story 2.8: --bisect and --keep-state runtime mechanics (Wave-Engineering/ccwork-testtarget#20)
• Story 2.9: Self-validation contract — red Tier 4 test for one Plan Plan: Beta — Wave-pattern reliability + autonomy hardening (operational backlog campaign B) #607 bug (Wave-Engineering/ccwork-testtarget#21)

References

• Sketchbook (input to /devspec): docs/automated-background-testing-sketchbook.md
• Parent Epic: Epic: Close the wave-pattern automated-testing gap (integration harness, daemon coverage, sub-agent contract enforcement) #626 — automated testing gap follow-up
• Origin: Plan Plan: Beta — Wave-pattern reliability + autonomy hardening (operational backlog campaign B) #607 (Beta) operational-failure debrief; "campaign-as-test-of-last-resort" diagnosis
• Memory files: lesson_destructive_test_homedir.md, lesson_doc_singular_branch_prefix.md, decision_platform_adapter_retrofit.md, WAVE_AXIOMS.md
• Open follow-up bugs (Tier 4 red-test fixture candidates): cc#629–bug(wave-pattern): GitLab wave::N labels and phases-waves.json issue lists can disagree — pick one as authoritative or surface drift #638, sdlc#425
• Operational policies: policy_wave_engineering_merge_config.md (merge-queue/auto-merge enrollment for harness repo)

[bakeb7j0]

[ledger D-001] /devspec §1.5 NG-01 · 2026-05-07T16:08:30Z
Decision: Tier 2 (per-MCP-tool roundtrip) and Tier 3 (skill-grain) coverage are out of scope for Plan: Gamma; deferred to a follow-up Plan as backfill on bisection pain.
Rationale: Plan #607 operator hand-recovery cost was paid at the e2e seam, not at the unit grain. Tier 4 e2e tests in this Plan incidentally exercise ~80% of MCP tools and ~60% of skills; the marginal value of explicit unit fixtures is recoverable later, when a Tier 4 failure proves expensive to localize.

— Loomweaver 🪶 (cc-workflow)

[bakeb7j0]

[ledger D-002] /devspec §1.5 NG-02 · 2026-05-07T16:08:30Z
Decision: Tier 5 chaos / fault-injection coverage is out of scope for Plan: Gamma; deferred to a follow-up Plan once the v0 runner shape proves out.
Rationale: Building chaos coverage on an unproven runner is premature — the runner itself is the load-bearing v0 deliverable. Once Tiers 1 + 4 ship and run nightly cleanly, deliberate failure injection becomes a tractable follow-up. Sketchbook §"Tier structure" makes the same argument; this entry locks it.

— Loomweaver 🪶 (cc-workflow)

[bakeb7j0]

[ledger D-003] /devspec §1.5 NG-03 · 2026-05-07T16:08:30Z
Decision: Tier 4 tests 4.2 (multi-flight parallel), 4.3 (multi-flight with file conflict), 4.4 (multi-wave dependency chain), 4.5 (multi-phase Plan), and 4.7 (KAHUNA on non-main base ref) are out of scope for Plan: Gamma; deferred to a follow-up Plan once 4.1 / 4.6 / 4.8 prove the v0 runner shape.
Rationale: v0 ships three platform-parameterized end-to-end tests covering single-flight + cross-repo + GitLab parity — sufficient signal to validate the runner's cross-platform fixture lifecycle and forensic-replay tooling before extending to richer orchestration shapes. The deferred tests are incremental coverage on a proven runner, not foundation work.

— Loomweaver 🪶 (cc-workflow)

[bakeb7j0]

[ledger D-004] /devspec §1.5 NG-04 · 2026-05-07T16:08:30Z
Decision: Epic #626 sub-issues #2 through #9 (daemon-process coverage, canonical-line enforcement, bus-path-scheme assertion, substring → exact-line audit, CHANGELOG-fragment guard, harness-backgrounding instrumentation, kahuna-parity preflight, pr_merge retry policy) are out of scope for Plan: Gamma; they ship as standalone Stories filed directly under Epic #626 via /issue feature --epic 626.
Rationale: Per Epic #626's body, those eight sub-issues are scoped narrowly enough to execute as single Stories — they do not require their own Plan tracking issues or /devspec walks. Plan: Gamma elaborates Epic #626 sub-issue #1 (the integration-harness foundation) only. Tests 4.1 / 4.6 / 4.8 in this Plan provide fixture infrastructure that sub-issues #8 (kahuna-parity preflight) and #9 (pr_merge retry) will validate against — complementary, not redundant.

— Loomweaver 🪶 (cc-workflow)

[bakeb7j0]

[ledger D-005] /devspec §1.5 NG-05 · 2026-05-07T16:08:30Z
Decision: Behavioral / model-fidelity testing — whether sub-agents follow WAVE_AXIOMS, whether prompt sections deter narrative drift, prompt regression, cost regression, and real-Discord-channel state — is out of scope for Plan: Gamma.
Rationale: These are observational concerns with no deterministic-CI surface. They require trend analysis on real campaigns, not unit assertions. Acknowledged as untestable surfaces in the sketchbook; addressed via post-campaign retrospectives and mcp.jsonl event-trend reporting, not by Plan: Gamma's harness.

— Loomweaver 🪶 (cc-workflow)

[bakeb7j0]

[ledger D-006] /devspec §1.5 NG-06 · 2026-05-07T16:08:30Z
Decision: Re-architecture of the wave-pattern itself is out of scope for Plan: Gamma; this Plan tests the existing pipeline.
Rationale: Inherits Epic #626's "Out of scope" position verbatim — closing the testing gap on the existing pattern is the goal, not redesigning it. Any architectural changes surfaced by harness findings become input to a separate Plan, not an in-band scope expansion.

— Loomweaver 🪶 (cc-workflow)

[bakeb7j0]

[ledger D-007] /devspec §2.1 TC-01 · 2026-05-07T19:14:13Z
Decision: Test harness runner stack is Python + pytest. The harness MUST be implemented in Python with pytest as the test framework.
Rationale: Pair-locked 2026-05-07. Bun was the alternative (matches the MCP fleet's language); Python chosen for pytest's mature fixture model and for forcing JSON-RPC-protocol-only access to MCP servers (TC-02) — a Bun runner could too easily import MCP server internals "for speed" and bypass the protocol surface that's where bugs actually appear.

— Loomweaver 🪶 (cc-workflow)

[bakeb7j0]

[ledger D-008] /devspec §2.1 TC-02 · 2026-05-07T19:14:13Z
Decision: MCP server interaction is JSON-RPC stdio only. Harness MUST drive each MCP server as a subprocess via the same JSON-RPC stdio protocol Claude Code uses; direct in-process import of MCP server code (Bun or Python bindings) is forbidden.
Rationale: The protocol surface is where bugs manifest in practice — the pr_comment-against-issue failure hit minutes ago during this session is exactly that class. Testing internals would mask bugs that only manifest at the protocol layer.

— Loomweaver 🪶 (cc-workflow)

[bakeb7j0]

[ledger D-009] /devspec §2.1 TC-03 · 2026-05-07T19:14:13Z
Decision: Filesystem isolation: tmpdir or env-override only. Tests MUST NOT touch real $HOME, ~/.claude/, ~/.config/claude-code/, or any path used by a live Claude Code session or another harness instance. All filesystem state lives under pytest-managed tmp_path or fixture-injected env-overridden directories.
Rationale: Source: lesson_destructive_test_homedir.md (mcp-server-discord token-eating-tests incident). Tests touching real homedir paths corrupt the operator's environment and can leak credentials.

— Loomweaver 🪶 (cc-workflow)

[bakeb7j0]

[ledger D-010] /devspec §2.1 TC-04 · 2026-05-07T19:14:13Z
Decision: Fixture artifact isolation by per-run-id prefix. Every Tier 4 fixture creates GitHub/GitLab artifacts (issues, branches, MRs, kahuna refs) with a unique run-id prefix. Teardown deletes by prefix only.
Rationale: Cross-run state leak is the silent killer of integration suites (sketchbook §"Architecture"). Per-run-id-prefixed creation + prefix-scoped teardown is the only shape that survives long-term unattended operation; flaky-tests-from-stale-state become indistinguishable from real bugs without it.

— Loomweaver 🪶 (cc-workflow)

[bakeb7j0]

[ledger D-011] /devspec §2.1 TC-05 · 2026-05-07T19:14:13Z
Decision: Fixture residency is Wave-Engineering/ (GitHub, private) and gitlab.com/testtarget/ (GitLab). All fixture repos / projects live in these namespaces only; no fixtures in cc-workflow, mcp-server-sdlc, or any production-org repo.
Rationale: Pair input 2026-05-07: BJ's GitHub free-tier cannot create new orgs, so private repos under Wave-Engineering/ are the available isolation. gitlab.com/testtarget/ is a pre-existing namespace BJ owns. Production repos are off-limits to fixture pollution.

— Loomweaver 🪶 (cc-workflow)