incomplete-edit costs about one extra iteration beyond what diff size predicts: find what distinguishes it at declare-done - #1944
Conversation
…+ RED tests (checkpoint)
…chema + analysis doc (checkpoint)
…egister run.sh test block
…field tests (verbatim changed_files, key-present, valid-falsy)
…-falsy/has assertions (post-shadow polish)
…-edit-costs-about-one-extra-iteration # Conflicts: # lib/test/modules/coverage-map.json # lib/test/run.sh # skills/retrospective/SKILL.md
…_version claim (already 3 on main) and clean-entry comment default wording Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
/prflow:review |
|
/prflow:review |
There was a problem hiding this comment.
Verdict: APPROVE (additive schema echo with matching tests; every acceptance criterion met; no findings)
Issue Compliance
Reviewed against issue #1827: incomplete-edit costs about one extra iteration beyond what diff size predicts — criteria from workpad; scope unchanged. Requirement-based checklist items are included in the verification results below.
Criteria came from the /prflow:implement workpad comment — this run's authoritative set, possibly narrowed from the issue.
No divergence between the workpad's criteria and the issue body's.
All four acceptance criteria are satisfied:
- AC1 — fields echoed + preserved; absent entry still cleans.
lib/clean-entry.jqaddsadditions/deletions/changed_files, each(.field // null), under the unchangedschema_version: 3;skills/retrospective/SKILL.mdechoes all three in both the output-schema block and the example construction, extending the "echo straight from the bundle — do not recompute" instruction to them. The producerlib/fetch-pr-context.shalready emits all three fields (verified against head), so "echoed from the bundle" holds. A bundle lacking the fields still cleans, defaulting each tonull. - AC2 — analysis doc.
docs/internal/incomplete-edit-cost-analysis.mdexists and records the rework metric (efficiency_runs[].iterations, max non-null per PR), its coverage (45/109 = 41.3% cohort; 175/489 store-wide), and the §2.3 sweep-selection correlation, concluding with counts that the durable records are insufficient to decide. Table arithmetic is internally consistent. - AC3 — metric never from
post_bot_commits. The doc's Method section states this explicitly and cites #1440 as the reason; no derivation uses that field. - AC4 — learnings byte-unchanged. No file under
.prflow/learnings/is touched by the commit range (verified).
Verification Checklist Results
0 failed, 0 inconclusive. The verifiable claims (producer emits the three fields in the shapes the consumer/tests assume; jq // treats 0/[] as truthy so valid-falsy values survive; schema_version 3 always emits the keys) were confirmed by the Phase-3 agents against head a64b426. Diff classification: engine_self_modifying (touches skills/** and lib/**) → full checklist profile.
Code Review Findings
No findings from any of the five dispatched Phase-3 agents (code-reviewer, silent-failure-hunter, comment-analyzer, pr-test-analyzer, requesting-code-review). The one executable change (lib/clean-entry.jq) is a fail-safe additive echo; the #1827 test block pins presence passthrough, absent→null defaulting, the has() key-always-present contract, changed_files verbatim-by-contents (not length), and valid-falsy (0/[]) passthrough — the classic //-coercion trap is explicitly guarded.
over-grade annotation: no finding flagged
truthfulness sweep: no finding promoted
intra-diff contradiction scan: no contradiction found
Prompt-surface edit routing evidence gate
The diff touches skills/retrospective/SKILL.md, so the DevFlow-repo routing gate applies. Discharged: issue #1827's workpad carries a Writing-skills evidence: marker with all four required dispositions explicit — skill-loaded=yes guidance-applied=yes pressure-scenario=no micro-tests=no. A recorded no never draws a finding on its own. The Verification evidence: advisory marker is also present (whole-suite CLEAN pass on a64b426), so the advisory clause is silent.
Shipped-prose cost audit
The skills/retrospective/SKILL.md addition (~9 lines) adds only what documents the three new fields — the schema entries, the echo-list extension, and the example-construction lines — each carrying the necessary "echo these from the bundle, do not recompute" instruction. No excess prose; no finding.
Verdict Criteria
No verification FAIL/INCONCLUSIVE, no self-contradicting-diff finding, no review-agent finding at or above the critical threshold, and no findings at all → APPROVE (rule 7).
Note: this standalone cloud review ran with the checkout on main; the reviewed head a64b426b was verified via git show <head>:<path> per the standalone PR-number routing. The Phase 0.6 stale-prose lint is disabled by config (prflow_review.stale_prose.enabled: false).
There was a problem hiding this comment.
Verdict: APPROVE with notes (small, purely-additive change; two low-value Suggestions, neither gating)
Issue Compliance
Reviewed against issue #1827: incomplete-edit costs about one extra iteration — criteria from workpad; scope unchanged. Requirement-based checklist items are included in the verification results below.
Criteria came from the /prflow:implement workpad comment — this run's authoritative set, possibly narrowed from the issue.
No divergence between the workpad's criteria and the issue body's.
All four acceptance criteria are satisfied:
- AC1 (fields echoed + preserved; absent-field entries still clean):
lib/clean-entry.jqaddsadditions/deletions/changed_fileswith// nulldefaults;skills/retrospective/SKILL.mdechoes them verbatim from the bundle; the producerlib/fetch-pr-context.shactually emits all three (additions/deletionsas numbers,changed_files=[.files[].path], an array), so the echo carries real data. Satisfied. - AC2:
docs/internal/incomplete-edit-cost-analysis.mdexists and records the rework metric (efficiency_runs[].iterations), its 41% cohort coverage (45 of 109), and the §2.3 sweep-selection correlation, concluding with a counts-backed insufficiency statement — the escape hatch AC2 permits. Satisfied. - AC3: the doc explicitly states the metric "is never derived from
post_bot_commits" (citing #1440); no table or finding is sourced from that field. Satisfied. - AC4: the diff touches no file under
.prflow/learnings/(verified against the merge-base); existing entries are byte-unchanged. Satisfied.
Verification Checklist Results
8 passed, 0 failed, 0 inconclusive — 0 via lite probe, 8 via agent.
✅ Passed items (8 of 8) — click to expand
- VC-1: additions/deletions/changed_files echoed from the bundle, preserved by clean-entry.jq, absent-field entry still cleans [lib/clean-entry.jq:133-137]
- VC-2: incomplete-edit-cost-analysis.md records metric, coverage, and counts-backed conclusion [docs/internal/incomplete-edit-cost-analysis.md]
- VC-3: rework metric never derived from post_bot_commits [docs/internal/incomplete-edit-cost-analysis.md:53]
- VC-4: existing .prflow/learnings/ entries byte-unchanged (diff touches none)
- VC-5: bundle producer emits all three fields [lib/fetch-pr-context.sh:888-890]
- VC-6: changed_files is a JSON array of paths, surviving verbatim [lib/fetch-pr-context.sh:48]
- VC-7: #1827 fixtures match the real bundle field names/types [lib/test/run.sh:166-188]
- VC-8: additive under schema_version 3 (no bump); keys always emitted (has() contract) [lib/clean-entry.jq:17,133-137]
Code Review Findings
🟡 Suggestion / Minor
- The end-to-end producer→
clean-entry.jqjoin is not pinned anywhere: the#1827block testsclean-entry.jqin isolation against hand-authored fixtures, and nothing asserts thatlib/fetch-pr-context.shemitschanged_filesas a path array (vs a scalar count). If the producer later regressed,clean-entry.jqwould silently default tonull/pass a wrong-typed value with no RED. The producer is pre-existing and unchanged by this PR, so this is a pre-existing-surface gap, not a regression this change introduces. (raised by 1/5 agents) - No test distinguishes an explicit
"additions": nullin the bundle from an absent field; under.additions // nullboth yieldnull, so the case is genuinely redundant — noted for completeness only, not recommended. (raised by 1/5 agents)
Notes
- Test evidence (CI, reviewed commit
a64b426b):lib + python tests: success, all five shards (monolith,python-pool,modules-pin,modules-large,modules-rest)success, andlint (shellcheck + actionlint + ruff): success. Authoritative per the engine ground-truth block. - Valid-falsy correctness: the
#1827tests pin that a real0and[]survive uncoerced (jq's//treats them as truthy) — the exact#312/#304off-switch bug class CLAUDE.md flags for//-based extraction, with a regression that fails closed against a naiveif .x thenrewrite. - Prompt-surface edit routing gate (repo policy): discharged — the linked issue #1827 workpad carries
Writing-skills evidence:forskills/retrospective/SKILL.mdwith all four slots recorded (skill-loaded=yes,guidance-applied=yes,pressure-scenario=no,micro-tests=no); all dispositions explicit, so the gate passes (a recordednonever draws a finding). - Verification-evidence advisory (repo policy):
Verification evidence:marker present on the workpad (cloud-tier run) — advisory clause silent. - Shipped-prose cost audit (repo policy): the
skills/retrospective/SKILL.mdadditions are a minimal, factual schema-field/echo edit (3 schema lines + a one-clause echo-list extension + 3 jq-example lines); every added word carries the instruction it names. No wasteful prose. - over-grade annotation: no finding flagged.
- truthfulness sweep: no finding promoted.
- intra-diff contradiction scan: no contradiction found.
- stale-prose lint disabled by config.
- Deferrals: no Scope-Acknowledged Findings block in the PR body.
- Dirty-tree backstop: disabled this dispatch (the pre-dispatch snapshot redirect was refused by the harness); the advisory review agents are read-only and made no working-tree change.
Verdict Criteria
- No verification checklist FAIL or INCONCLUSIVE.
- No self-contradicting-diff finding (every diff-added claim verified true against HEAD
a64b426b). - No review-agent finding at or above the configured
criticalthreshold — only two Suggestions, both below threshold and one of them pre-existing surface. - → APPROVE with notes.
Summary
additions,deletions, andchanged_filesfrom the context bundle, making diff size a durable, reproducible control for future analysis instead of requiring a liveghre-fetch.incomplete-editcost investigation as an internal doc page whose answer is that the current durable records are insufficient to decide whether the category is predictable at declare-done.schema_version3 (already 3 onmainsince A retrospective entry records nothing about what evidence the analyst had, so diff-present and diff-absent entries pool indistinguishably #1829); no schema bump, and existing.prflow/learnings/entries are byte-unchanged.Changes
Retrospective schema (
skills/retrospective/SKILL.md): Documentsadditions,deletions, andchanged_filesin the Stage A output schema, the echo-straight-from-the-bundle prose, and the jq example construction.Entry cleaner (
lib/clean-entry.jq): Preserves the three size fields from the bundle, each defaulting tonullvia//(so a producer lacking them still cleans without error, and the keys are always present underschema_version3).Analysis doc (
docs/internal/incomplete-edit-cost-analysis.md, new): Records the cohort method, the rework metric (efficiency_runs[].iterations, neverpost_bot_commits) and its ~41% coverage, and concludes the durable records cannot decide the question, naming what a future pass would need.Tests (
lib/test/run.sh): Adds a#1827block asserting the size-field cleaning contract.Coverage map (
lib/test/modules/coverage-map.json): Registers issue 1827 asunmodularized.Changeset (
.changeset/issue-1827-incomplete-edit-size-fields.md, new):patch/Added.Resolves
Resolves #1827
Test Plan
#1827block inlib/test/run.sh:additionsanddeletionssurvive cleaning when present (120, 34).changed_filessurvives verbatim by contents and order (not merely by length).null, with the keys still present (has()pins theschema_version3 contract).0/[]survives and is not coerced tonull, andchanged_filesstays an array.Visual Changes
N/A
Breaking Changes
None
Prompt-surface size
Derived at
a64b426bc74f60c31561c5ba6946da06d6948c96against merge-base9a2147e221859cbbecff25bf23421487c7cd579b(origin/main). Covered: tracked*.mdunderskills/,agents/,.prflow/prompt-extensions/.skills/retrospective/SKILL.mdGenerated via /prflow:implement (v2.34.14, claude-opus-4-8, low)