Repository navigation
Description
Problem
A configured reviewer can return a valid clean findings object and be recorded as complete even when it omitted the independent agents required by its installed skill. Later manual audits discover the missing execution, after additional workflow work or review cycles have already been consumed.
Observed with the retained dk-workflows 1.1.24 runtime in the October 27–29 audit of scylladb/python-driver#1149 and scylladb/alternator-client-python#157.
Observed sequence and evidence
- In PR #1149's earlier fifth cycle, the configured reviewers returned no findings, but Toolkit had not launched its required specialists. The iteration reported a coverage limitation and stopped; CI also had a separate unresolved test failure.
- In PR Keep workflow execution reports private across all skills #157's first cycle, Toolkit reported completion without running its required specialists. The omission was retained in the audit history, and subsequent cycles explicitly checked execution.
- Other sampled runs had insufficient retained nested telemetry to independently certify every pass. That uncertainty is not proof that an agent was omitted and should remain distinct from verified nonexecution.
The runner instructs the provider to follow its skill and validates the returned findings schema. The completed response contract does not require method-specific execution evidence before accepting {"findings":[]}.
| def run_skill_reviewer(name, skill, handoff_path, checkout): |
The observed omissions are supported by retained session audits. No new reviewer was launched for this report, and no omission is inferred solely from unavailable telemetry.
Expected behavior
- Require retained execution evidence for the specialist/independent passes required by the selected installed skill before claiming complete coverage.
- Bind that evidence to the reviewer invocation, selected skill/version, pinned scope, and review session.
- Validate completion in the runner; a provider's clean findings object alone must not prove its internal method ran.
- Distinguish verified missing execution from unavailable evidence, provider failure, usage exhaustion, and completed work.
- Preserve original results and findings; do not repair provenance by retroactively marking an unperformed pass complete.
- Add regressions where a provider returns valid clean JSON while omitting a required pass, and where execution evidence is missing, stale, mismatched, or valid.
Related: #47 concerns concurrent execution of configured reviewers. This issue concerns verification of the required work inside each selected reviewer.
Activity