Skip to content

ci: add the distribution verification workflow and its verifier - #2139

Merged
The01Geek merged 4 commits into
mainfrom
policy-update/distribution-verify
Aug 29, 2026
Merged

ci: add the distribution verification workflow and its verifier#2139
The01Geek merged 4 commits into
mainfrom
policy-update/distribution-verify

Conversation

@The01Geek

@The01Geek The01Geek commented Aug 29, 2026

Copy link
Copy Markdown
Owner

Add the distribution verification workflow

This is the first of two pull requests in the PRFlow distribution migration. It
adds exactly two files and changes nothing else.

File Purpose
.github/workflows/distribution-verify.yml Verifies a release candidate's bytes. Becomes the required status check on main.
.github/verify-release.py The verifier it runs. A byte-identical projection of the canonical verifier held in the development repository.
lib/test/run.sh Inventory updates only — see below.

Why a third file

Adding a workflow obliges updating every inventory in this repository's own suite
that enumerates the workflow set, in the same change:

  • #582 — the three-way partition must cover every .github/workflows/*.yml.
    distribution-verify is plugin-internal: never shipped to a consumer.
  • #1402 — the never-shipped set is the copy-loop complement, so it gains the
    same entry.
  • #936verify-release.py names devflow-review.yml in its
    forbidden-workflow set. That is a surviving reference to a deleted workflow and
    belongs in the machine-consumed inventory of such references.

No test logic changes; only three inventory literals. Those literals are the
drift-proof record of workflow membership, so leaving them stale would silently
widen or narrow what the audit covers.

Why this lands separately, before the release

The verifier must exist on the protected base branch before it judges anything.
If it arrived in the same pull request as the release, that release would be
installing its own judge — and the workflow's central rule is that a candidate
cannot modify the thing evaluating it.

That rule is enforced mechanically: on every pull request the workflow compares its
own two files against the base branch and fails if either differs. An intentional
change to the verifier therefore lands as its own preparatory pull request, judged
by the previous verifier, exactly like this one.

What it does

  • Copies the tree without .git and verifies that copy. Candidate content is
    treated as data: nothing from it is imported, sourced or executed.
  • Checks the release payload's SHA-256 manifest, required members, forbidden
    paths, file modes, line endings, symlinks, Windows-invalid filenames, case
    collisions, size budget, provenance integrity, and relative-link resolution.
  • Needs no secrets and requests only contents: read.

Effect on this repository today

None. It adds a workflow that passes on the current tree and takes no action.
Nothing about the plugin, its version, its install path, or consumer behaviour
changes here.

The01Geek and others added 4 commits August 29, 2026 12:01
Lands before the release so the judge exists on the protected base branch: a
release candidate must carry verifier and workflow bytes identical to base, so a
pull request cannot modify the thing evaluating it. The workflow enforces that
comparison itself.

Adds two files and changes nothing else. Requests only contents: read, needs no
secrets, and treats candidate content as data - it imports, sources and executes
nothing from the tree under review.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… non-candidate tree

The base-comparison step could never be satisfied by the pull request that adds
the judge, and a tree without .release/ metadata - the normal state of main
before the first generated release - was treated as a failed candidate.

Bootstrap is accepted only when no part of the judge is on base and the branch
is policy-update/*; a release/* branch missing provenance still fails hard.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ntories

Adding a workflow file obliges updating every inventory that enumerates the
workflow set, in the same change:

  #582   the three-way partition must cover every .github/workflows/*.yml;
         distribution-verify is plugin-internal, never shipped to consumers
  #1402  the never-shipped set is the copy-loop complement, so it gains the
         same entry
  #936   verify-release.py names devflow-review.yml in its forbidden-workflow
         set, which is a surviving reference to a deleted workflow and belongs
         in that machine-consumed inventory

These three literals are the drift-proof record of the workflow membership;
leaving them stale would silently widen or narrow what the audit covers.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…rify

The #1402 set is asserted at two sites - once over the real sources and once
with /dev/null sources to prove the print flag never consults them. Both carry
the membership literal, so both move together or the second goes red alone.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@prflow-implementer

Copy link
Copy Markdown

/prflow:review

@github-actions

github-actions Bot commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

PRFlow Review — PR #2139

Status: ❌ REJECT
Diff profile: engine_self_modifying + has_new_types + detect_all_audit → full checklist, Phase 3.1.5 completeness-critic forced
Run: View run
Reviewed HEAD: fd61203
Last updated: Phase 4 — verdict REJECT
Stale-prose lint (Phase 0.6): 0 STALE (all UNRESOLVABLE — cloud HEAD predates merge; informational only)

Blueprint

  • Classify diff (Phase 0.5)
  • Generate verification checklist (Phase 1) — 18 items
  • Verify checklist (Phase 2) — 17 passed, 1 failed (VC-12), 0 inconclusive
  • Review agents (Phase 3) — 6 agents returned; completeness-critic ran (audits complete)
  • Aggregate & verdict (Phase 4) — REJECT
  • Run complete — everything this run owed

Findings (live)

  • [comment-analyzer + final-pass, corroborated ×2; also Phase-2 VC-12] documented_falsehoodverify-release.py:40-45 byte-budget comment claims lib/test (~14.8 MB) and .prflow/learnings (~7.5 MB) "either blows this ceiling on its own", but MAX_BYTES = 12 MiB and .prflow/learnings measures ~7.46 MB — below the ceiling. False universal against HEAD. (rationale comment; behavior-inert)
  • [pr-test-analyzer, Critical] test_gapverify-release.py (leak/allowlist boundary, lines 103-249) ships with zero unit tests; the secret-leak detection is a silent-failure security control.
  • [pr-test-analyzer, Important] test_gap — structured-input parsers check_digests/check_source_json/check_manifest (306-390) have no adversarial input-shape matrix.
  • [pr-test-analyzer, Important] test_gapcheck_links path-resolution / template-suppression (259-303) untested.
  • [pr-test-analyzer, Important] test_gap — workflow judge-immutability + provenance-gating shell logic (distribution-verify.yml:492-553) untested.
  • [type-design-analyzer, Suggestion] type_designFailure(Exception) (130-131) is dead/vestigial (never raised/caught; module uses an errors-accumulator).
  • [type-design-analyzer, Suggestion] type_designWIN_BAD_CHARS uses set not frozenset; required/forbidden disjointness invariant unenforced (56-133).
  • [silent-failure-hunter, Suggestion] unhandled_exceptioncheck_links (271) reads read_text without the OSError guard its sibling scanners apply (also 313, 375); an unreadable file aborts with a traceback (loud, not fail-open).
  • [silent-failure-hunter, Suggestion] securitycheck_leaks (239-249) decodes with errors="ignore", so a secret in a non-UTF-8 region is silently dropped before the pattern scan (fail-open; small blast radius).

Completeness-critic (Phase 3.1.5): FORBIDDEN_WORKFLOWS ⊇ independent workflow enumeration; REQUIRED_DIRS matches vendor-slice.sh; check_manifest/check_digests are genuine bidirectional set-equality — no finding.

Verdict: REJECT — a diff-added comment states a false universal (self-contradicting diff), and the new 434-line security-boundary verifier ships with zero tests

Issue Compliance

No related issue found — requirement compliance not checked. (PR body cites #582/#936/#1402 as context, not Resolves; branch policy-update/distribution-verify carries no issue-N pattern.)

Verification Checklist Results

17 passed, 1 failed, 0 inconclusive — 0 via lite probe, 18 via agent.

  • VC-12: FAIL — the byte-budget comment claims lib/test (~14.8 MB) and .prflow/learnings (~7.5 MB) "either blows this ceiling on its own", but MAX_BYTES = 12 MiB and .prflow/learnings measures 7,459,833 bytes (< 12,582,912). The "on its own" universal is false for the second payload. [.github/verify-release.py:40-45]
✅ Passed items (17 of 18) — click to expand
  • VC-1: workflow invokes the verifier as python3 .github/verify-release.py --candidate and argparse requires only --candidate [.github/workflows/distribution-verify.yml]
  • VC-2: judge-diff step reads pull_request.base.sha/head.ref and uses them under fetch-depth: 0 [.github/workflows/distribution-verify.yml]
  • VC-3: artifact step branches on head.ref || github.ref_name and falls through to ref_name on push [.github/workflows/distribution-verify.yml]
  • VC-4: _936_EXPECTED gains .github/verify-release.py, asserted equal to the surviving-reference set [lib/test/run.sh]
  • VC-5: _582_INTERNAL gains distribution-verify; three-list concatenation asserted exhaustive-and-disjoint vs on-disk basenames [lib/test/run.sh]
  • VC-6: both --print-never-shipped-set literals gain distribution-verify in sorted position [lib/test/run.sh]
  • VC-7: check_source_json requires the seven fields, 40-hex source_commit, .release/-excluded counts [.github/verify-release.py]
  • VC-8: check_digests two-space split, covers emitted set minus manifest [.github/verify-release.py]
  • VC-9: check_manifest tab-split, # comments, emitted==declared [.github/verify-release.py]
  • VC-10: verifier never imports/execs candidate content (candidate is data) [.github/verify-release.py]
  • VC-11: LEAK_ALLOWLIST suppresses only the exact (path, match) pair; break follows only an append [.github/verify-release.py]
  • VC-13: REQUIRED_DIRS (7) matches vendor-slice.sh's cp root set [.github/verify-release.py]
  • VC-14: FORBIDDEN_WORKFLOWS enumerates dev workflows, excludes devflow.yml/devflow-implement.yml [.github/verify-release.py]
  • VC-15: self-contained; only stdlib imports [.github/verify-release.py]
  • VC-16: WIN_RESERVED = CON/PRN/AUX/NUL + COM1-9 + LPT1-9; stem-uppercase check [.github/verify-release.py]
  • VC-17: bootstrap refuses a partially-present judge for any branch [.github/workflows/distribution-verify.yml]
  • VC-18: git diff --quiet BASE_SHA HEAD -- <judge> detects self-modified judge under merge-ref checkout [.github/workflows/distribution-verify.yml]

Code Review Findings

🔴 Critical

  1. A diff-added rationale comment states a false universal: .github/verify-release.py:40-45 claims lib/test (~14.8 MB) and .prflow/learnings (~7.5 MB) "either blows this ceiling on its own", but MAX_BYTES = 12 MiB and .prflow/learnings measures ~7.46 MB — below the ceiling, so it does not blow it alone (raised by 2/6 agents; also Phase-2 VC-12 FAIL) [self-contradicting-diff carve-out: .github/verify-release.py] [inertness: first-conjunct=held limb-one=held limb-two=not-held; evidence: verify-release.py ships byte-identical to the public distribution repo per its own docstring, so the comment reaches external readers of published source; no tool parses it (Phase-0.6 stale-prose lint 0 STALE; not a declaration marker); the byte-budget mechanism (MAX_BYTES compared against the walked total in check_budget) is unchanged whether the sentence is true or false → limb-two prevents inertness, carve-out governs]. Fix: reword to state that lib/test alone exceeds the ceiling while learnings contributes on top of the normal payload.
  2. The new 434-line verify-release.py — a security boundary whose whole purpose is to stop secret leaks, path leaks, digest tampering and internal-file inclusion from reaching the public release — ships with zero unit tests; the check_leaks/LEAK_PATTERNS/LEAK_ALLOWLIST control (lines 103-249) is entirely unexercised (raised by 1/6 agents) [suspected over-grade: shape 3 — single-source from pr-test-analyzer; observable fail-direction is missing coverage with no runtime fail-direction on its own, milder than the Critical label — advisory only, does not demote].

🟠 Important / Major

  1. The structured-input parsers check_digests/check_source_json/check_manifest (306-390) have no adversarial input-shape matrix (malformed/missing/wrong-type/duplicate/valid-falsy), which CLAUDE.md's parser-hardening convention requires (raised by 1/6 agents).
  2. check_links path-resolution, ${}… template-suppression, and the hand-rolled ../. normalizer (259-303) are untested; a normalizer regression that resolves everything as "present" would silently disable the link check (raised by 1/6 agents).
  3. The workflow's judge-immutability and provenance-gating shell logic (distribution-verify.yml:492-553) is untested; standard PRFlow practice treats workflow run: blocks as testable shell, and the branch table (bootstrap-allowed/refused, partial-present-refused, release-without-provenance-refused) should be pinned (raised by 1/6 agents).

🟡 Suggestion / Minor

  1. Failure(Exception) (130-131) is dead/vestigial — never raised or caught; the module signals via an errors accumulator and exit code. It ships verbatim into the public projection. Delete it or give it a real, enforced role (raised by 1/6 agents).
  2. WIN_BAD_CHARS uses a mutable set where its siblings use frozenset; the required/forbidden disjointness invariant (56-133) is unenforced (a cheap module-load assertion would catch a contradictory future edit) (raised by 1/6 agents).
  3. check_links (271) reads read_text without the try/except OSError guard its sibling scanners (check_leaks 239-242, check_modes_and_endings 226-228) apply, so an unreadable .md/.mdx aborts the verifier with a traceback and loses the accumulated-errors report — a loud failure, not fail-open (also at 313, 375) (raised by 1/6 agents).
  4. check_leaks (239-249) decodes with errors="ignore", so a secret straddling an invalid-UTF-8 run is silently dropped before the pattern scan — a fail-open direction, with a small blast radius given the ASCII/UTF-8 text corpus (raised by 1/6 agents).

over-grade annotation: finding flagged — Critical #2 (shape 3, advisory).
truthfulness sweep: finding #1 (Critical) confirmed as demonstrated documented_falsehood against HEAD and routed into the self-contradicting-diff carve-out.
intra-diff contradiction scan: no additional contradicting absolute-claim/limitation pair found.
Completeness-critic (Phase 3.1.5): the introduced audits (FORBIDDEN_WORKFLOWS coverage, REQUIRED_DIRS vs vendor-slice.sh, check_manifest/check_digests bidirectional set-equality) are complete w.r.t. the independent workflow-directory / cp-root / directory-listing signals — no finding.

Verdict Criteria applied

Run telemetry

Per-phase calls/tokens/wall_clock_s: unavailable (not instrumented this run). Durable --persist record not written (read-only cloud review profile).

Subagent effectiveness trace

Iteration 1

  • Diff profile: engine_self_modifying + has_new_types + detect_all_audit
  • Phase 3 agents dispatched: 6
  • Checklist verifiers: 0 lite, 18 agent.
  • Effectiveness signal: verdict contribution (standalone review applies no fixes) — 3 of 6 agent(s) contributed
  • Agent verdicts:
    • prflow:code-reviewer — null
    • prflow:comment-analyzer — corroborating
    • prflow:pr-test-analyzer — unique-effective
    • prflow:requesting-code-review — corroborating
    • prflow:silent-failure-hunter — noise
    • prflow:type-design-analyzer — noise

@The01Geek
The01Geek merged commit ff7f6f4 into main Aug 29, 2026
10 checks passed
@The01Geek
The01Geek deleted the policy-update/distribution-verify branch August 29, 2026 18:38

@prflow-reviewer prflow-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: REJECT — full report in PR comment

The complete review report (checklist results, findings, details) is in the
PRFlow Review progress comment on this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant