docs(decisions): 0252 — define dispersion and the two-week decline, ship the trend co-gate observe-only (#4766) - #5259
Conversation
No preview deploy
|
|
review-doc: FAIL @ c450c20 — changes-requested Reviewed-head: @ c450c20 Class fan — §CP — both axes derived. Path clause: clean (live CODEOWNERS covers only Read the PR head (§HEAD): every file under review sourced from The findingsF1 — BLOCKING. The 0.05 calibration sentence states the opposite of the criterion it justifies.
Both halves are false against the ADR own arithmetic.
Why this blocks rather than being a copy nit. This is the only sentence justifying the only free constant in the record, and #4766 AC5 asks that the ADR make #4680 discriminating unit test authorable. A builder authoring that test from §3 would encode "one case regressing does not flag" as a Remedy (author call, one of two):
F2 — the series-ordering key
|
|
review-code: FAIL @ c450c20 — changes-requested Reviewed-head: @ c450c20 Class fan — §CP — both axes derived. Path clause: clean — live CODEOWNERS covers Read the PR head (§HEAD): every file under review sourced from No build/test run, and none owed. The diff adds zero executable lines — two prose surfaces inside a code root. There is no behavior to exercise, so the gate is a read of the diff plus a ground-truth check of every factual claim it makes against the code at head. The findingF2 — the
|
…hip the trend co-gate observe-only (#4766)
…ring key (#4766) Repair round 3 on the two blocking findings against ADR 0252 §3. F1 — the calibration sentence stated the inverse of the criterion. `0.95 − 0.90` is the WHOLE 0.05, not half of it, and with an inclusive `≥ 0.05` clause one regressing case in a twenty-case corpus lands exactly on the threshold and DOES flag. Resolved by making clause 1 strict (`> 0.05`), which is the reading the section's own "deliberately strict" stance and clause 2's strictness already imply: 0.05 is the corpus's smallest step, and a threshold that fires on one step is not a threshold. Strictness alone was not enough. The boundary sits exactly on a value the input can produce, and in IEEE-754 that value straddles the constant — two arithmetically identical one-case regressions compute 0.04999999999999982 and 0.050000000000000044, on opposite sides of 0.05 under either polarity. So the criterion now rounds the half-mean difference to 4dp before comparing, which is ~1e12 coarser than the double error and 1/500 of a case. Added the boundary row to the worked table — the fixture a builder is most likely to guess wrong. F2 — the window was ordered by a `recordedAt` the `Scorecard` type does not carry. Rather than invent it: §3 now states the criterion is specified ahead of its input and names #4680's date pin as the blocking dependency, with `recordedAt` as the spelling to use (the word the spend ledger already stamps). Carried to the `two-week decline` glossary row and the eval README so a builder meets the same caveat at whichever surface they open. The trend stays disarmed: no code path reds on it, and none is added here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
c450c20 to
584c2e5
Compare
|
review-doc: advisory — blocking-set PR (§CP — approval-gated) Reviewed-head: @ 584c2e5 Repair round 3, re-gated fresh at the new head. Both prior verdicts ( §CP re-derived on both axes at this head. Path axis: clear — none of the three changed paths matches the canonical F1 — the calibration arithmetic and the strict/inclusive fork — [PASS]The design call is right. Clause 2 was already strict; §3's stated stance is to prefer a missed decline over a false flag while the gate is unarmed; and The prose now matches the spec exactly. Re-derived:
Worked table — all five rows re-derived independently, all correct:
Row 4 is the fixture a builder would otherwise encode backwards, and it is now present, correct, and flagged in prose as the one row a reader is most likely to guess wrong. The table alone discriminates declining / flat-noisy / boundary / insufficient — the discriminating unit test #4680 requires is authorable from it. The 4dp rounding rule (added beyond what the FAIL asked) — [PASS]The straddle is real, and I reproduced it rather than taking the claim. Enumerating every one-case regression over a twenty-case corpus with three points per half (old half 4dp settles it. Both coarseness claims check out. Max observed deviation from exact is about Two implementers converge. Adding this beyond the FAIL's ask was the right call: strictness alone would have left the boundary non-deterministic, which is the same defect one layer down. F2 — the ordering key — [PASS]Verified in source at this head, not taken from the repair note:
Cannot be misread as claiming the field exists today. The first mention ("Order by the scorecard's Regression checks — [PASS]
Acceptance criteria (issue #4766)
Deviations — reviewed, acceptedThe scope-narrowing Deviation (four criteria ask for amendments on the bodies of #4678 / #4680 / #4681; this lane was authorized for #4766 only, so the amendment text was filed as #5258 instead) is declared, justified, and correctly disposed. Routing around the cross-issue-write refusal would have been the wrong call; surfacing it fail-loud with the exact text staged is the right one. The unruled-judgment Deviations (picking the three constants; writing the arming bar as a proposal rather than a ruled bar) sit inside the founder's explicit delegation to engineering, and the one thing that delegation does not cover — when the gate may start blocking — is correctly left as an open fork rather than decided here. Accepted, unchanged from the prior rounds. Non-blocking notes — NOT grounds for a fourth roundNone of these blocks. Recorded so a later editor has them; the ADR is accepted as-is.
VerdictPASS (advisory). Both prior FAILs are fully discharged, and the repair's unrequested extension (the 4dp rounding rule) is independently verified as correct, necessary and well-calibrated. No blocking finding remains; the three notes above are non-blocking and explicitly not grounds for a fourth round. This verdict authorizes nothing on its own (§CP — approval-gated). A Verdict-written: 2026-08-10T03:25:59Z |
|
review-code: advisory — blocking-set PR (§CP — approval-gated) Reviewed-head: @ 584c2e5 Repair round 3, re-gated fresh at the new head. The prior Class fan. §CP re-derived on both axes. Path axis: clear — none of the three paths matches the canonical No executable code changed. The diff is The disarm — no code path reds on the trend — [PASS]Verified independently at this head, not taken from the repair note. Grepping
No other hit anywhere in the package. The claim holds exactly as stated: no code path can red on the trend, because no code path reads it. The disarm is structural (the read does not exist yet), not a flag that could be flipped by accident. This is stronger than the ADR's own observe-only ruling requires. Glossary freshness (Step 3c) — [PASS]
The F2 contradiction is resolved. The Module README (
|
usirin
added
the
status:cp-banked
2 participants
Two values the fabrika grading chain leans on —
dispersionand a "two-week decline" — were named in the spec and never defined, and the second one was wired into a merge-blocking red. This PR writes both down as ADR 0252. It also rules that the trend co-gate ships observe-only: it reports its answer and cannot red a PR until the criterion has been watched against a real recorded series, which is the founder guardrail on this ticket.Fixes #4766
What lands
.decisions/0252-grading-chain-dispersion-and-decline-criterion.md— the record.dispersionismin(passed, runs − passed)over the graded axis's five runs: one integer, units of runs, 0–2 at five runs, derived from the five per-run verdicts alone. It rides the head-bound eval-result artifact as{"runs": 5, "passed": 3, "dispersion": 2, "verdict": "pass"}—runsandpassedtoo, so a reader re-derives it instead of trusting it.(stage × surface × model)cell's pass-rate series across the committed scorecards: the trailing 14 days anchored on the newest point, split into two 7-day halves with at least 3 points each;decliningiff the half-mean drop (rounded to 4dp) is strictly greater than0.05andmax(new) < min(old); otherwisesteady; fewer points isinsufficient-data, which is never a pass. A worked five-row table is in the ADR so the discriminating unit test can be authored from it.below-barreds on the ruled 90% bar alone; the trend answer rides the same verdict as an advisory line and never changes an exit status. Arming it is a separate later ADR, ruled on the observe-only record.packages/fabrika-cli/src/eval/README.md— a short pointer section so a builder inside the module finds the arithmetic..glossary/TERMS.md— canonical rows fordispersionandtwo-week decline.Why the numbers are what they are
0.05is the whole distance from a healthy 0.95 to the ruled 0.90 bar, and — the reason the comparison is strict — it is exactly one case in a twenty-case corpus. A threshold that fires on the corpus's smallest step is not a threshold, so the line is drawn just above it: one case regressing issteady, two cases (0.10) isdeclining. The comparison rounds the half-mean difference to 4dp first, because0.05is a value the input lands on exactly and in IEEE-754 that value straddles the constant — two arithmetically identical one-case regressions compute0.04999999999999982and0.050000000000000044. The non-overlap clause is what separates drift from scatter — mean drop alone is cleared by a noisy flat series often enough to be useless, and it is also what makes the required "declining flags, flat or noisy does not" test writable. All three constants (0.05, 3 points per half, 14/7 days) are explicitly a starting point calibrated against the observe-only record before they may block anything.Needs a ruling
Two forks this PR deliberately does not settle, because the ticket carries no ruling for them:
below-barregains the trend clause at arming, or stays a single-condition red permanently. This PR removes it for now on the guardrail's authority. The end state is the arming ruling's call.Nothing here re-opens the 90% bar or the co-equal status of the trend gate (#4637 ruling 2). The trend still flags before the 90% line is crossed — flagging is what observe-only does. What is deferred is only its authority to block.
Deviations
dispersionand "two-week decline" are undefined, and one gates a merge-blocking red #4766's criteria ask for the decision to land as amendments on the bodies of graded axis: the reused grader-agent judgment path at 5 runs and a median #4678 and committed scorecards, the trend co-gate, and the model-churn re-run contract #4680, and for the ruled bar as the merge gate: 100% floor, 90% graded co-gated on trend, cost at or below baseline #4681'sbelow-barto cite the criterion.dispersionand "two-week decline" are undefined, and one gates a merge-blocking red #4766 only.dispersionand "two-week decline" are undefined, and one gates a merge-blocking red #4766 per this lane's standing control, so the residue lives on Propagate ADR 0252 onto the grading-chain issue bodies (#4678, #4680, #4681) #5258 rather than on a still-open fabrika grading chain:dispersionand "two-week decline" are undefined, and one gates a merge-blocking red #4766 — flagged here so the reviewer can judge that call.review-docF1 offered two remedies — make clause 1 strict, or keep≥and rewrite the calibration sentence to say one case does flag.0.05is a value the input lands on exactly, and in IEEE-754 it straddles the constant — two arithmetically identical one-case regressions compute0.04999999999999982and0.050000000000000044, on opposite sides under either polarity. Without a stated rounding the criterion's answer at its own boundary depends on which cases moved.dispersion" is wrong. That correction is recorded in the ADR and repeated in Propagate ADR 0252 onto the grading-chain issue bodies (#4678, #4680, #4681) #5258's amendment text; the issue body itself is untouched for the reason above. Disposition: no action needed beyond Propagate ADR 0252 onto the grading-chain issue bodies (#4678, #4680, #4681) #5258.Nothing else. Docs-only diff:
pnpm typecheckgreen over the whole workspace,pnpm lint:worktreea clean skip (no biome-handled files changed),decisions-index validateclean. ADR number 0252 was picked after enumeratingorigin/mainplus every open PR's.decisions/additions (0235, 0237, 0250, 0251 are in flight).