Harvest: add an eval before adopting behavior-changing candidates

[rathko]

The problem

Harvest is report-only, which is right. But when a candidate changes how the system behaves (Algorithm mechanics, a skill's workflow logic, hooks, routing policy), the only thing standing between "sounds great" and "adopted" is a review. A review can tell you a design is internally broken. It cannot tell you the new mechanism beats the one you already run. Without a measurement step, harvests slowly fill the system with plausible ideas that measure worse than what they replaced. That is where bloat comes from: every exciting blog post becomes a candidate, and enthusiasm is the only gate.

What happened to me

I harvested multi-model "plan fusion" into my LifeOS-derived fork (from the Devin Fusion, OpenRouter fusion-beats-frontier and aider architect/editor posts): generate two plans with different frontier models at the commitment boundary, then have a judge synthesize one. On paper it looked like an obvious win. It survived a 10-concern adversarial cross-model design review after I folded every concern into a v2.

Then I ran an offline A/B before adopting:

• 7 historical high-effort ISAs as the task set
• ground truth from ISA Changelogs: conjectured -> refuted_by entries, the assumptions each real run only discovered were wrong late. A plan scores when it anticipates those pitfalls up front
• cost parity (3 model calls per arm), authorship-blind pairwise judging with position swap
• decision rule written down before any results existed: adopt at 0.65 win share or better, reject under 0.45

Fusion won 1 of 14 judgments and anticipated 3 fewer real pitfalls than the current plan -> advisor critique -> revise loop. Rejected. Seven tasks is a small sample and plans were judged rather than executed, so I treat the number as directional. Without the eval I would have shipped a measurably worse mechanism into the most leveraged part of the Algorithm, and the review had already blessed it.

Proposal

Add a Step 7 to skills/Harvest/Workflows/Harvest.md, between Report and Ingest (Ingest becomes Step 8):

Step 7 — Eval before adoption (behavior-changing candidates only)

For any candidate whose adoption would change system behavior — Algorithm mechanics, a skill's workflow logic, hooks, routing/delegation policy — the Recommended action proposes eval-first: an offline comparison against the current mechanism, with a decision rule written down before results exist, and blind judging where feasible. Write eval criteria to the standard of ALGORITHM/eval-guide.md (the 3-question test). Past ISA Changelog conjectured → refuted_by entries make useful test labels where they exist. Adoption still requires {{PRINCIPAL_NAME}}'s explicit go-ahead after the eval.

Cosmetic or non-measurable candidates (format rules, doc grafts) skip the eval — note the skip in one line. When unsure whether a candidate is behavior-changing, treat it as behavior-changing.

Why: review catches bad designs; only evals catch worse-than-baseline designs. A mechanism can survive adversarial review and still lose a blind comparison to the status quo it was meant to replace.

And one Gotchas bullet in SKILL.md:

Behavior-changing candidates need an eval, not just a review. If adopting a candidate would change how the system behaves (Algorithm mechanics, skill workflow logic, hooks, routing policy), the Recommended action proposes an offline comparison against the current mechanism, with a decision rule written down before results exist — review catches bad designs; only evals catch worse-than-baseline designs. Cosmetic candidates skip with a one-line reason.

Why I think this fits

The pieces already exist in LifeOS. ALGORITHM/eval-guide.md has the 3-question test for judging criteria. ISA Changelogs already record refuted conjectures, which make decent test labels. The gate only binds behavior changes, so a normal harvest stays as light as it is today. Markdown only, no code.

[danielmiessler]

Landed in adapted form. The Harvest mining workflow now notes that for behavior-changing candidates (prompt-shape swaps, doctrine rewrites, routing changes) the recommendation may propose an offline eval with the decision rule stated up front, citing the eval guide. One deliberate difference from the proposal: it's elected, not mandatory. A standing rule here is that evals are a discretionary resource, never a required gate, and adoption already sits behind an explicit human go-ahead. Your plan-fusion worked example (1/14) was the best argument for the elected version. Credited in the workflow and the README.