Skip to content

PLAN.md

Latest commit

 

History

History
789 lines (582 loc) · 113 KB

PLAN.md

File metadata and controls

789 lines (582 loc) · 113 KB
Raw

Research plan

This file is the living plan: where Kev stands, what runs next and the criteria decided before the runs, and open questions. Completed plans stay as records under History (the Qwen3.5 port, done 2026-09-20, is there). The active round is Round 4 below (current-architecture levers, registered 2026-09-22); the proposal after it is PLAN_27b.md (Qwen3.8-27B, question-side LoRA, long documents), gated on round 4's items 4.8 and 4.12. Everything older is kept below under History, dated.

Where we stand (2026-09-22)

  • Family: Kev-0.8B / 4B / 9B on Qwen3.5 bases, one recipe (decision-v7, LoRA r=16, lr 1e-4 / 5e-5 / 5e-5) plus the 2026-09-21 dates + unknowable delta. Locked test, out of domain: 0.684 / 0.837 / 0.852; Jev 0.857 on the development items. Pre-delta weights at Hub tag v7-base; Qwen3 checkpoints published as the previous generation.
  • Gap to Jev (pre-delta Kev-9B, transfer-v4 dev, 4.5 pp overall; 3.5 pp after the delta): knowledge (MMLU 0.74 vs 0.90; MMLU-Pro 0.545 vs 0.84) — the untrained base scores the same, so this is base capacity; date arithmetic (deadline 0.72 vs 0.93) — any LoRA fine-tune on our format erodes the base's skill (0.82 → 0.72), while the readout is intact (giving the model the day count yields 0.93–1.0, issue #8); calibration — Brier 0.291 vs 0.211, coverage at ≤ 5 % error 0.53 vs 0.70, confident errors 7.5 % vs 3.7 %; robustness — assertion-style Noul instructions ("The customer sounds angry.") drove Kev-4B to 0.79 vs Jev 0.91 on scienthoon's tickets.
  • Tools now available: --init_from delta fine-tunes from a released checkpoint (minutes, not hours); --data JSONL for custom records; transfer-v9 (MMLU-Pro, buried, unknowable) and the SemIf / scienthoon external suites; coverage-at-error-budget and unknowable metrics in kev.benchmark.
  • Budget: Modal metered $405.94 on 2026-09-22 (round-3 registration read $392.14); authorization $1,000 total, of which round 4 below may use $60 before a fresh billing check.

Round 4 — current-architecture levers before the 27B (registered 2026-09-22)

Why this round exists. Two external data points (analysis of SemIf and Solomon, 2026-09-22) moved the priorities: zero-shot Qwen3.8-27B beats trained Kev-4B on SemIf's authored suite (0.958 balanced vs 0.847), so base capacity is now the main lever for external numbers — but Kev's shipped temperature does not transfer to WANLI (Kev-9B ECE 0.131, 21/132 errors at p ≥ 0.9, runs/kev-9b-wanli-v1) while SemIf's per-workload T fixed a comparable ECE (0.21 → 0.07) without training. Every item below is cheap at the current sizes, and any recipe finding transfers to the 27B run at a hundredth of its cost. Nothing that landed between the night-2 sign-off (4f8110a) and this registration changes training: the refactors were bit-identical (PRs #18, #21), the three loss flags default to 0, the temperature is eval-only — so no retrain of 0.8B / 4B / 9B is needed on that account.

Already tried with negative results, not repeated: loss modifiers (round 3), per-group temperatures (round 2 #2a), DeltaNet-frozen LoRA (#5), the assertion delta alone (#4), 35B-A3B (#1/#8).

Tier 0 — no training. Reporting changes need no gate; each option below has one.

# item run adopt if (pre-registered) cost
4.1 Per-workload calibration path. kev.calibrate --rows <rows.json>: restore raw logits (rows record inference_temperature), fit one T with group-disjoint OOF folds, print raw / shipped-T / OOF-T rows with bootstrap CIs. Commit rows.json for the external runs so anyone can refit. local, $0 Reporting. Cards gain a "calibrated to this workload (OOF)" row for WANLI and TypeSafe next to the raw row. $0
4.2 KEV_DATE_FACTS as serving default. Evidence: dev 0.797 → 0.820 (4B), 0.822 → 0.828 (9B), no harm observed on transfer-v4 (round 3, H1). Unknown on external states. one benchmarks pass: 9B + 4B with --date_facts on semif-v1, scienthoon-v1, wanli-v1, typesafe-v1 Flip the default (opt-out KEV_DATE_FACTS=0) only if no external suite drops by more than 0.5 pp accuracy and no confident-error rate rises by more than 0.5 pp; otherwise stays opt-in and the negative is recorded. Benchmark headline rows stay raw either way. ~$3
4.3 Paired-CI incumbent rule in kev.autoresearch. Today incumbent()/score() pick by the transfer point estimate; with one seed and ~650 questions that chases noise. code, $0 A challenger replaces the incumbent only if its record-clustered paired bootstrap on transfer accuracy has lower bound ≥ −1 pp and point estimate above the incumbent; the CI is written to leaderboard.jsonl. No new training. $0
4.4 Rotation-averaged Choice at test time: score every cyclic rotation of the options, log-mean the option probabilities. Kev's pointer head is content-addressed, so this should do little; permutation.mean_max_delta says it is not zero. benchmark --rotations, 9B + 4B on transfer-v4 dev Becomes a serving option (never default) if transfer accuracy improves ≥ +0.5 pp with a paired CI lower bound ≥ 0 or the permutation flip rate halves; otherwise recorded as evidence the design already handles order. ~$2
4.5 Two-checkpoint averaging (round-3 C6, never run): mean probabilities of the two v7 seeds per size (9B s0/s1, 4B s2/s3, 0.8B three seeds) from saved transfer/rows.json. local, $0 Informational unless coverage at ≤ 5 % error improves ≥ +5 pp or Brier improves with a paired CI excluding zero; then a second du seed (~$2) and a serving option that doubles cost. $0
4.6 bf16 / MLX flip rate at scale. The parity claim rests on 40 records. scripts/mlx_parity.py on the full development partition, 4B + 0.8B (Mac) Reporting: flip rate and max Δp published in the cards. $0
4.7 Re-run scripts/calibrate_checkpoint.py on Kev-9B and Kev-0.8B so head.pt["temperature_fit"]["cross_validation"] exists for all three (same T; audit field only). Republish head.pt only. local, $0 Reporting. $0

Tier 1 — training, deltas or single trials, each attacking a measured gap. Selection on development partitions only; one locked read per adopted candidate; paired, record-clustered bootstraps throughout (kev.metrics.paired_bootstrap).

# question run adopt if (pre-registered) cost
4.8 Does a post-trained dense 9B keep its date arithmetic through Kev training? Open question since night 2: every Base checkpoint erodes deadline (0.82 → 0.72 at 9B), the post-trained 35B-A3B kept it (0.88 → 0.95). Qwen/Qwen3.5-9B (pinned c202236235762e1c871ad0ccb60c8ee5ba337b9a, same hybrid layout as the Base) has never been probed. (a) zero-shot probe, plain + SemIf prompt, transfer-v4 + transfer-v9; (b) one v7-recipe trial, lr 5e-5, seed 0; (c) if (b) passes, the dates + unknowable delta on top. Probe gate for (b): plain or SemIf zero-shot ≥ 0.73 on transfer-v4 dev (the Base scored 0.729) — a post-trained base that reads the format worse is not worth a trial. Candidate gate for (c) and a locked read: deadline raw ≥ 0.85 (Kev-9B-du raw 0.80), transfer dev within 1 pp of 0.822 or above, MMLU not lower by more than 1 pp, coverage at ≤ 5 % error not lower by more than 3 pp, held-out pairs ≥ 0.75. Promotion needs the locked read ≥ 0.852 − 1 pp with no external suite worse by more than 1 pp. The card must say the base is post-trained. probe ~$4, trial ~$7, delta ~$2
4.9 Ambiguity soft targets (round-3 C5, never run). 16 of Kev-9B's 26 confident transfer errors are PAWS; 59 % of all errors sit on the three noisy-label sources. Where an open teacher disagrees with the public label at ≥ 0.6, train toward a label/teacher mixture instead of the hard label — the materialize soft-target mechanism the unknowable records already use. Teacher: the untrained post-trained Qwen3.5-9B (SemIf prompt) or the 4B one already scored (runs/probes/qwen35-4b-semif-transfer-v4); never Jev. teacher pass over the decision-v7 training partition's public rows (~$3); two deltas (9B, 4B) from the released checkpoints, replay 2000, lr 2e-5, 1 epoch; T re-fitted on each trial's own dev rows Coverage at ≤ 5 % error on transfer-v4 dev (served probabilities) ≥ +5 pp over the released checkpoint with accuracy within 1 pp (paired CI lower bound ≥ −1 pp), AURC no worse, unknowable share ≥ 0.9 on transfer-v9 ≤ 0.05. Report every trial regardless. ~$8
4.10 Reliability head on the unused 968-record decision-v7/calibration partition: logistic on (p_max, margin, entropy, K, type) re-ranking confidences. Solomon shipped and then dropped one; a genuine experiment. fit locally on saved rows; evaluate OOF on transfer-v4 dev + WANLI AURC or coverage at ≤ 5 % error improves with a paired CI excluding zero on both evaluation sets; otherwise recorded negative. $0
4.11 Self-distillation 9B → 0.8B: Kev-9B's served probabilities as soft targets for a 0.8B delta on the same training partition (our own outputs; the no-Jev rule is untouched). one delta ~$1.50 Transfer dev ≥ +2 pp over Kev-0.8B-du (0.643 seed) with paired CI lower bound > 0 and coverage not lower; then one locked read. ~$2
4.12 Long-state delta at 4B (de-risks PLAN_27b.md B2 before any document set exists): the buried generator at 1k / 2k / 4k-token states, MAX_STATE lifted for the delta, short-state suites re-scored. one delta ~$3 Informational: report the short-state cost (transfer within 1 pp is the hope) and the long-state gain. Anything beyond this belongs to the 27B plan. ~$3

Order and stop rules. 4.1 → 4.3 → 4.7 → 4.2 + 4.4 (one Modal pass) → 4.5 + 4.6 → 4.8(a) → 4.8(b) → 4.9 → 4.10 → 4.11 → 4.12. Total bound ≈ $35 plus probes; fresh modal billing summary before 4.8(b) and before 4.9. If 4.8 produces a candidate, that is the retrain — one size, one trial, one locked read — not a from-scratch family run. The 27B plan waits for 4.8 and 4.12: a post-trained 9B that holds its arithmetic changes which 27B to pin and drops the question-side-LoRA ablation in priority; the long-state cost sizes B2. Every result, met or not, is written below this section with its run directory.

Round 4 results

(filled in as each item completes)

Round 2 autoresearch (2026-09-20 → 21) — done; results below

Ordered by expected value. Each trial is either a delta (warm start from the released checkpoint, new records mixed with a replay sample of decision-v7, lr 2e-5, 1 epoch) or a probe (no training). Selection on development partitions only; one locked read per adopted candidate.

# question run cost adopt if (pre-registered)
1 Does a bigger MoE base fix the knowledge column? Zero-shot probe of Qwen3.5-35B-A3B-Base on transfer-v4 + v9 (H200) $6 Proceed to a Kev-35B-A3B trial only if MMLU-Pro ≥ 0.70 and overall ≥ the 9B base's 0.729.
2 Can calibration be bought without accuracy? (a) per-(type, K) temperature fitted on dev; (b) unknowable records with uniform targets as a delta from Kev-9B / Kev-4B (a) $0, (b) 2 deltas ~$8 Adopt if coverage@5 % error improves ≥ +5 pp on transfer-v4 dev with accuracy within 1 pp of the released checkpoint; for (b) also unknowable share ≥ 0.9 falls (4B: 0.19 → ≤ 0.10). Temperature is reported as a separate row, never folded into raw numbers.
3 Date arithmetic by construction (issue #8): date-bearing families rendered with relational day counts and a date_facts field; opt-in date_facts request preprocessor 2 deltas ~$8 Adopt the renderings if deadline ≥ 0.85 with the preprocessor and raw deadline does not fall, accuracy elsewhere within 1 pp. Report raw and preprocessed separately.
4 Assertion-style Noul instructions 1 delta each at 9B / 4B ~$8 Adopt if scienthoon angry ≥ 0.85 at 4B with transfer-v4 within 1 pp.
5 Where does the eroded arithmetic live? Retention ablation: 4B from scratch, LoRA on attention + MLP only (DeltaNet projections frozen) 1 trial $5 Informational; if deadline ≥ 0.65 raw, DeltaNet-frozen becomes a candidate recipe for a follow-up.
6 Combined delta (2b + 3 + 4) from Kev-9B and Kev-4B 2 deltas ~$10 Same rules as its parts; this is the promotion candidate if the parts pass.
7 Third-party comparability: Kev-9B / 4B on ekzhang's 1,000-question MMLU-Pro sample (seed 42); GPQA-diamond as an eval-only source if the dataset is accessible $3 Reporting only.
8 If #1 passes: Kev-35B-A3B, decision-v7, lr 5e-5, LoRA on attention + DeltaNet + shared expert, routed experts frozen, H200/B200 ~$40 Ship as a hosted tier only if transfer-v4 dev ≥ Kev-9B + 2 pp and MMLU-Pro ≥ 0.70; one locked read.

Rules: accuracy comparisons are record-clustered paired bootstraps on the same items; "within 1 pp" means the point estimate. Deltas replace a released checkpoint only after the locked read confirms no regression there. Every adopted change is written into the model cards with the criteria outcome, met or not.

Deferred, in order: MLX serving for the hybrid on Mac (the release's one regression: 0.78 s vs 0.17 s at 4B); multilingual slices; high-cardinality column; GGUF/browser path.

Results (2026-09-21, 02:30)

Spend tonight ≈ $105 (probes $14, deltas 12 × ~$1.5, dense $5, benches ~$15, locked reads $8, five failed or duplicated 35B launches ~$18, two 35B trials $24). Budget authorization: ~$475 of $500 used. Working notes were kept in scratchpad.txt (deleted; in git history up to f3bae08).

# result verdict
1 Qwen3.5-35B-A3B-Base zero-shot: transfer-v4 0.720 (9B base 0.729), MMLU 0.82, MMLU-Pro 0.590, deadline 0.75, rules worse. Gate failed (needed MMLU-Pro ≥ 0.70). No base-MoE trial.
1′ Qwen3.6-35B-A3B (post-trained) zero-shot with the SemIf prompt: 0.812 = trained Kev-9B; MMLU 0.85, deadline 0.88, emotion 0.62; rules weak (0.62 / 0.56). Plain prompt 0.726. Gate passed (≥ 0.76) → two trials.
8 Kev on Qwen3.6-35B-A3B (v7 recipe, --weights_dtype bf16, LoRA 21 M on attention + DeltaNet + shared expert, routed experts frozen, H200, 136–158 min, 72 GB peak): lr 5e-5 → dev 0.869, transfer 0.823 (+1.2 pp vs Kev-9B [−3.0, +5.5]; −3.2 vs Jev [−7.7, +1.2]); lr 2e-5 → 0.819. Profile: deadline 0.93 / 0.95 (Jev 0.93; the post-trained base's date skill survives training), MMLU 0.81 / 0.80 (+7 over Kev-9B, −9 vs Jev), held-out pairs 0.84 / 0.88 (Jev 0.86); but TweetEval 0.75 / 0.71 (Kev-9B 0.78), Emotion 0.57 / 0.54, or_not rule 0.78 (0.88), confident errors 8.8–9.1 %, coverage@5 % 0.50 / 0.40 (0.53). transfer-v9 (lr 5e-5): knowable 0.778 (Kev-9B 0.771), MMLU-Pro 0.550 (Kev-9B 0.545, Jev 0.840 — the base's MMLU gain did not carry to 10-way MMLU-Pro), buried 0.68 (0.74), unknowable share ≥ 0.9 0.14 (0.05). Not shipped. Fails the pre-registered bar (≥ Kev-9B + 2 pp); lands where Kev-9B + dates + unknowable already is (0.822 dev / 0.852 locked) with worse calibration and 8× the memory. It does answer the question: a bigger post-trained base buys knowledge and dates and loses noisy-label classification; it does not reach Jev on this suite. Checkpoint kept on the volume (night2-36b6/00-trial-0); a 35B + dates/unknowable delta is a possible follow-up, not a priority.
2a Single temperature T ≈ 2.0 fitted in-distribution does transfer OOD on the Qwen3.5 family: Kev-9B Brier 0.291 → 0.267, ECE 0.105 → 0.039, confident errors 7.5 % → 3.2 % (Jev 3.7 %), accuracy and coverage unchanged. Per-(type, K) temperatures are worse (OOD Score items have a K the in-distribution fit never saw; per-group scaling scrambles the cross-group confidence ranking). Adopt as a reported calibrated row; KEV_TEMPERATURE in kev.serve. The coverage criterion was ill-posed for a global T (monotone). Grouped T rejected.
2b Unknowable records with uniform targets (delta): share of evidence-free items answered at ≥ 0.9 → 0.00 at both sizes (from 0.05 / 0.19), controls unchanged. Coverage@5 % on transfer-v4 dev: 9B 0.53 → 0.48, 4B 0.54 → 0.57. Confidence criterion passed decisively; coverage criterion failed at 9B. Partial.
3 Date renderings (delta) + date_facts preprocessor: deadline 9B 0.72 → 0.80 raw → 0.90 with the preprocessor; 4B 0.55 → 0.60 → 0.82 (0.85 in the combined delta). The preprocessor alone on the released models: 0.75 / 0.68 — training to bind the fact is what makes it work. Passed (≥ 0.85 with preprocessor, raw not lower, accuracy within 1 pp).
4 Assertion-style Noul (delta): scienthoon angry 4B 0.794 → 0.821 alone, 0.859 in the combined delta; 9B 0.911 → 0.924. Side effect: raises confidence on evidence-free items (unknowable share 0.05 → 0.25 at 9B) and adds a few p = 1.00 errors on Noul tasks (tweet, PAWS) that cut coverage. Passed only in combination; the combination's coverage cost is partly this.
5 Dense ablation (LoRA on attention + MLP only, DeltaNet frozen, 4B seed 1): deadline 0.53 → 0.53, transfer 0.800 → 0.780. Freezing the recurrent layers does not preserve date arithmetic. Not a candidate.
6 Combined deltas, locked test (one read each): 9B + dates + unknowable 0.852 OOD (+1.8 pp [+0.8, +2.9] over Kev-9B's 0.837), Brier 0.237, deadline 0.88, pairs 0.81, coverage@5 % 0.62 (from 0.66); 9B + all 0.851; 4B + dates + unknowable 0.837 (+1.0 [−0.1, +2.1]), Brier 0.255, coverage 0.68; 4B + all 0.828 (neutral). Promotion candidates: the dates + unknowable deltas at both sizes, published as Hub branch night2-du (main untouched, awaiting sign-off). Costs to state in the cards: coverage@5 % −4 pp at 9B, scienthoon ECE +3 pp, MMLU-Pro −3 pp at 9B.
7 ekzhang's 1,000-question MMLU-Pro sample: Kev-9B 0.511, Kev-4B 0.468, Kev-8B (Qwen3) 0.488 (8 questions over the 384-token state limit counted wrong). Jev 0.829; untrained one-token Qwen3.6-35B-A3B 0.588; his $5 SFT ≈ 0.71. Reporting. Knowledge is base-bound; the 3.6 trial is the only lever. GPQA-diamond is gated — needs the account to accept terms.

Decisions taken (2026-09-21, morning, signed off)

  1. Promoted the dates + unknowable deltas to the main Hub revisions of kev-9b, kev-4b and kev-0.8b (the 0.8B delta ran in the morning: locked test 0.668 → 0.684, +2.2 pp [−0.8, +5.5], Brier 0.473 → 0.460). Pre-delta weights at tag v7-base; release tarballs *-v7-base.tar.gz. Cards state the costs (coverage@5 % −4 pp at 9B, scienthoon ECE +3 pp, MMLU-Pro −3 pp at 9B).
  2. Calibration built into the checkpoints (2026-09-21, later the same day): head.pt carries a temperature fitted on the trial's development rows (9B 2.30, 4B 2.14, 0.8B 2.41) and the pointer head applies it in eval mode, so every loader serves calibrated probabilities by default; KEV_TEMPERATURE=1.0 gives raw logits. Out-of-domain ECE 0.106 → 0.042 (9B), confident errors 8.7 % → 4.0 % (Jev 3.7 %); accuracy unchanged. Coverage at ≤ 5 % error is not moved by a temperature — that needs training-side calibration (PLAN_27b.md §5).
  3. date_facts preprocessor shipped opt-in (KEV_DATE_FACTS=1), documented as preprocessing with separate numbers.
  4. The Qwen3.6-35B-A3B checkpoint stays a research artifact.

Round 3 autoresearch — audited execution on research/calibration-audit

Execution order approved: metric audit → failure audit → matched loss screen → replicate a promising candidate → fresh final audit → decide whether a larger-model study is justified. No published weights or production deployment will be changed by this round. The binding machine-readable protocol is experiments/calibration-audit-protocol.json; it is frozen before new training results.

Measurement and failure audit

  • scripts/calibration_audit.py re-scores saved development predictions into a new report, leaving the originals unchanged. The old coverage metric could split equal-confidence ties and gave 0.94 or 0.00 for the same synthetic predictions under a row permutation. Version 2 admits whole tie groups. Jev's observed coverage changes from 0.704 to 0.695; the saved Kev values are unchanged (4B 0.573, 9B 0.447 after approximate temperature replay).
  • Coverage is a non-additive statistic: the paired bootstrap now resamples complete record groups and recomputes the curve, rather than bootstrapping per-row coverage values. The wide intervals on this small set motivate reporting the entire risk–coverage curve and AURC, not claiming a deployment guarantee from one empirical 5% cutoff. AURC uses a documented right-step integral over complete confidence groups. See kev/metrics.py and its regression tests in tests/test_research.py.
  • The previous statement that temperature cannot reorder confidence was incorrect. Positive temperature preserves each question's argmax, but may reorder top probabilities between multiclass questions, including equal-K questions. New local evaluations preserve raw logits and record effective temperature; historical probability replay is explicitly marked approximate because it applies a floor to rounded/saturated probabilities.
  • High-confidence-error share divides by all questions; error among accepted answers divides by accepted questions. At p_max ≥ 0.9, saved Kev-9B has 26/656 = 4.0% of all questions wrong but 6.8% error among accepted answers. These are not interchangeable calibration claims.
  • The AI-assisted failure audit inspects 26 confident errors and 26 matched correct controls. It finds executable rule errors, omitted facts, role-binding mistakes, ambiguous renderings, and possible annotation inconsistencies. No label is changed, and this is not human adjudication. PAWS is an adversarial human-judged reading benchmark; difficulty and teacher disagreement do not prove label noise. Cross-entropy is a proper scoring rule, not an established root cause of these failures. Guo et al. motivates calibration measurement, not a causal diagnosis of Kev.

Small, matched screen

Use the pinned Kev-4B parent from the protocol, seed 11, one epoch, lr 2e-5, effective batch 8. All four arms see the exact same 3,425 frozen training records, augmentations, ordering, and number of optimizer updates. A re-evaluated unchanged parent and a standard-CE continuation control are both required.

arm loss on hard-labelled questions motivation
CE control ordinary cross-entropy separates extra training from changing the loss
smoothing CE against (1−0.05) one_hot(y) + 0.05/K limited regularization, not assumed equivalent to temperature; Müller et al.
CE + Brier CE + 0.5 × sum of squared probability error proper-scoring alternative; Gneiting & Raftery
focal (1−p_y) × CE (gamma 1) empirical hard-example weighting; no promised calibration gain; Mukhoti et al.

Existing soft-target records use the same soft-target CE in every arm. Temperature is fitted from raw logits on decision-r3/calibration, never on development or final-test labels. Report both raw and recalibrated metrics, per-source results, and all failed trials. Teacher relabelling, the entropy-penalty grid, full retraining, and ensembles are deferred until there is evidence to justify them.

Screening gate: at least +5 percentage points of tie-aware micro coverage against both CE control and recalibrated parent; micro accuracy no more than 1 point worse; no source more than 5 points worse; AURC no worse. This screening gate is exploratory, not a significance claim. Advance at most one recipe, ranked by coverage, then AURC, NLL, and fixed arm order. If none passes, stop and leave the fresh test unscored.

Replication gate: best loss versus matched CE controls at seeds 12 and 13 on 4B and 9B, within the remaining budget. Require positive coverage differences at both seeds and ≥5 points mean gain, the same accuracy/AURC constraints, unknowable high-confidence share ≤0.05, and intact-control accuracy within 2 points of the parent. Write the chosen checkpoint hashes before any final read. Record-group bootstrap intervals do not measure training-seed variability; report seeds separately.

Fresh final audit and spend control

scripts/freeze_calibration_audit.py freezes decision-r3 and transfer-r3. The latter contains 580 new threshold-calibration records and 1,260 new final-test records, including separate unknowable/control diagnostics. The builder excludes recorded prior normalized states and public-source origins, preserves generated groups, and verifies zero final-test overlap with training, calibration, and development. This does not establish absence from foundation-model pretraining. No fresh-test predictions have been read at registration.

A qualifying recipe is evaluated once against its pinned parent and matching CE control. Select acceptance thresholds using the fresh threshold-calibration partition, then freeze them and apply them to the final set without re-selection. Recommend promotion only if the final coverage gain is ≥5 points against both controls with paired 95% CI lower bound >0, accuracy CI lower bound ≥−1 point, AURC no worse, and fixed-threshold empirical risk ≤5% on at least 100 accepted knowable questions. Report failure rather than choosing another candidate after reading this set. Previous test partitions are now historical regression evidence, not untouched confirmation.

Modal reports $392.14 metered at registration; this supersedes the previous unverified $475 estimate. The user subsequently authorized $1,000 total, before any new training runs. Keep the initial stage within $60; reserve further spending only for evidence-backed follow-ups, with a conservative $600 additional ceiling and a fresh billing check before each stage. Experimental gates are unchanged by the budget increase. Bound launches using the actual requested CPU, memory and GPU limits; do not redeploy kev-research, cancel other jobs, overwrite outputs, move release tags, or publish checkpoints. The possible $10,000 credit grant is not authorized spend. The larger-model work in PLAN_27b.md remains gated.

Round 3 screening result — stopped at the registered gate

All five jobs completed under calibration-screen-4b-s11-r2. Each trained arm used exactly 3,899 records including augmentations, 429 optimizer steps, and 639,399 forward tokens from the same 3,425-record corpus. The initial startup failed before training because the optional HF-secret dependency was declared differently locally and remotely; its five calls were cancelled with explicit approval, the environment was corrected, and the failed attempt is recorded separately. The production app was not redeployed.

Full comparison and paired intervals, concise outcome, risk–coverage curves, and scripts/review_calibration_screen.py. All values below are on the same 656 clean development questions, after fitting each checkpoint's temperature on the same independent calibration partition using raw logits. No test labels were used for fitting or selection. The generic runner's legacy NLL-based candidate flag is not this protocol's promotion gate; the authoritative screen selected no candidate.

arm accuracy coverage at ≤5% empirical error AURC ↓ Brier ↓ ECE ↓
unchanged Kev-4B, recalibrated 0.7973 0.5762 0.0580 0.2653 0.0478
CE continuation control 0.7973 0.5320 0.0625 0.2737 0.0519
label smoothing, epsilon 0.05 0.8018 0.0061 0.1001 0.2879 0.0671
CE + Brier, weight 0.5 0.7973 0.5259 0.0628 0.2744 0.0561
focal, gamma 1 0.7988 0.5015 0.0606 0.2678 0.0433

Decision: no candidate advances. Focal improves ECE slightly but not selective coverage or AURC. Smoothing slightly improves accuracy while harming ranking: AURC difference versus parent +0.0422 [95% CI +0.0228, +0.0651]. CE continuation also worsens selective performance, confirming why the matched control was necessary. These are results for one seed, one epoch and one setting per loss; they do not show that these losses fail universally.

As registered, no replication, fresh threshold-calibration read, final-test read, publication, or larger-model run follows this negative screen. The 1,260-record final panel remains unscored and available for a future preregistered candidate. The reasonable next question is a targeted data/capability experiment (for example, controlled role-binding counterexamples from permitted training sources), not an expanded loss grid or automatic relabelling of PAWS. It needs its own registration before spending.

Workspace metering initially showed $403.87 after the pass, then revised to $398.32 on the final check: about $6.18 above the $392.14 starting reading, subject to billing lag and workspace attribution. Both observations are retained in the outcome summary; neither is an exact per-trial invoice. The authorized ceiling is $1,000, not a target to spend. The successful screen's conservative function-execution admission bound was $18.92, excluding startup/storage. No historical benchmark file or published checkpoint was overwritten.

Training-data investigation after the negative screen

Analysis first, no training launched. Where the current checkpoints fail on the rule tasks (runs/binding-diagnostic-v1/report.json, scripts/build_binding_diagnostic.py):

  • Every Kev-4B rule error on transfer-v4 development involves an elapsed (date-difference) atom: 0.78 on structures with one, 1.00 on the 64 without; with KEV_DATE_FACTS=1 0.94. The 564 compositional training records with elapsed atoms never state a day count. Errors are not near the threshold (they occur at a +10-day margin too), so this is absent arithmetic, not off-by-one.
  • Kev-9B's ten rule errors sit in four generated groups (three renderings each), all confident accept on reject labels, all with match ("X is the same person as Y") or elapsed atoms. On the existing rows 9B scored 0/6 when a compared name recurred elsewhere in the case — but those six items are two groups.

A fresh eval-only diagnostic (560 records, code labels, new seed, disjoint from every frozen suite; inference only, ~$2) tested both readings with real sample sizes:

stratum (n) Kev-4B Kev-9B
match true (120) 0.992 0.992
mismatch, clean (120) 0.975 0.967
mismatch, decoy name in another role (120) 0.917 0.967
mismatch, second matching pair shares a name (120) 0.967 1.000
elapsed rule, dates only (40) 0.650 0.750
same cases with the day count stated (40) 1.000 0.975

H1 (dates) confirmed: 14 (4B) and 9 (9B) paired cases flip from wrong to right when the day count is stated; none flip back. The models already use a stated count; what they cannot do is subtract. That makes it a serving question, not a training-data one: training on day-count renderings would not change the plain case, and the earlier ablations showed fine-tuning erodes rather than teaches the arithmetic. The decision to take is whether KEV_DATE_FACTS becomes the serving default (development accuracy 0.797 → 0.820 at 4B, 0.822 → 0.828 at 9B, no observed harm elsewhere) — a product choice, still reported as preprocessing.

H2 (name-co-occurrence shortcut) not supported: at 9B the decoy strata equal the clean stratum. The 0/6 was two correlated groups — exactly the small-sample trap the audit warned about. Kev-4B shows a small effect (0.917 vs 0.975, 9 of 10 errors answer accept), worth at most ~1 pp on rule tasks; decoy-augmented training data is not a priority.

Not pursued: PAWS-style role binding in natural text. There is no permitted programmatic source that yields reliable labels for swapped-role paraphrases, and PAWS itself stays eval-only; this would need human-labelled data and its own registration.

Superseded initial round-3 proposal (retained for the research record)

The draft below predates the metric/failure audit. Its causal claims, teacher identity, budget and timing estimates, temperature-ordering argument, and test-selection procedure are superseded by the registered execution protocol above; they are not instructions for this run.

Why. After the built-in temperature, Kev's probabilities have the right scale — Kev-9B out-of-domain ECE 0.042, confident errors 4.0 % (card; Jev 0.049 / 3.7 % on the same items, runs/jev-transfer-v4/report.json) — but not the right order. Coverage at a ≤ 5 % error budget, the share of decisions that can be accepted in confidence order before the accepted set exceeds 5 % error (kev/metrics.py: coverage_at_error, the metric from AbdelStark/jev-benchmarks), is 0.45 at 9B and 0.57 at 4B vs Jev's 0.70. A temperature is monotone within a question, so it cannot move this (scripts/temperature_groups.py: coverage 0.53 → 0.52 at T = 2.0). The ceiling at Kev-9B's accuracy (0.822) is 0.86 if confidence ranked correctness perfectly; the gap to Jev is ordering, not accuracy.

Where the ordering fails (served probabilities, transfer-v4 development, runs/night2-9b-du/00-trial-0/transfer/rows.json): 26 of 656 answers are wrong at ≥ 0.9, and 16 of them are PAWS (adversarial paraphrase pairs); at 4B, 10 of 17. Across all errors, 59 % are on the three noisy-label sources (TweetEval, PAWS, Emotion), which are 37 % of the items. The training loss is hard-label cross-entropy (kev/train.py: question_loss), which asks for certainty on items that are genuinely ambiguous — the textbook cause of over-confidence in fine-tuned classifiers (Guo et al., 2017).

What to try — each is a flag on question_loss, run as a delta from the released checkpoint (--init_from, replay 2000, lr 2e-5, one epoch: ~15 min / ~$1.50 at 9B, round-2 recipe), then the temperature re-fitted with scripts/calibrate_checkpoint.py because every loss below changes the logit scale.

# change mechanism reference cost
C1 Label smoothing ε ∈ {0.05, 0.1} soft target (1−ε) on the label, ε/K elsewhere; known to improve calibration, mostly by shrinking confidence uniformly — a baseline that may behave like the temperature Müller, Kornblith & Hinton, 2019 2 deltas × 2 sizes
C2 Focal loss γ ∈ {1, 2} down-weights already-confident correct items so the gradient concentrates on hard ones; shown to yield calibrated networks without post-hoc scaling Mukhoti et al., 2020 2 × 2
C3 Cross-entropy + Brier term λ ∈ {0.5, 1} Brier is a proper scoring rule with bounded penalty for confident mistakes; the ordinal RPS term already in the trainer (--ord_w) is its ordered cousin Gneiting & Raftery, 2007 2 × 2
C4 Confidence penalty β ∈ {0.05, 0.1} adds −β·H(p) to the loss: rewards entropy where the label does not dominate Pereyra et al., 2017 2 × 2
C5 Ambiguity soft targets on the public sources where an open teacher (Qwen3.5-9B instruct, SemIf prompt — our strongest untrained readout, 0.747 on transfer-v4, runs/probes/qwen35-4b-semif-transfer-v4) disagrees with the label at ≥ 0.6, train toward a mixture of label and teacher instead of the hard label; the same mechanism as the unknowable records (evals/night2, kev/data.py: materialize target). Targets the actual cause; teacher outputs are open-weight, never Jev's. Hinton et al., 2015 (soft targets); our round-2 unknowable result half a day of data work + 1 × 2
C6 Two-checkpoint averaging at serve time averaging two seeds' probabilities improves calibration reliably; doubles serving cost Lakshminarayanan et al., 2017 $0 (existing seeds)

Pre-registered rules.

  • Metric: coverage at ≤ 5 % error on transfer-v4 development, served probabilities (after re-fitting T on the trial's own development rows, never on transfer or test). Secondary: Brier, confident-error rate, unknowable share ≥ 0.9 on transfer-v9 (must not rise above 0.05).
  • Adopt a change if coverage improves by ≥ +5 pp with accuracy within 1 pp of the released checkpoint (paired, record-clustered bootstrap, kev.metrics.paired_bootstrap); report every trial regardless.
  • If no delta clears the bar, one full retrain at 9B with the best-looking loss (~$7, 90 min) before concluding that calibration needs the full run rather than a touch-up.
  • Winners get one locked read and replace the released checkpoint under the same name, with the raw and served columns in the card.
  • Expectation stated in advance: partial — 9B 0.45 → 0.55–0.60. Closing to 0.70 probably needs accuracy gains too (coverage and accuracy are coupled).

Cost and time. ~20 deltas ≈ $30, ~4 h wall; C5's data step half a day; locked reads and republish the next morning. Fits the remaining round-2 budget; does not draw on the 27B credits (PLAN_27b.md).

Open questions

  • Why does LoRA fine-tuning on classification-shaped data erase multi-step latent computation (dates) while leaving recall (MMLU) intact? Freezing the DeltaNet layers does not help (trial 5), and on the post-trained 3.6 base the skill survives (0.88 → 0.95), so the erosion is specific to Base checkpoints. Layer-wise LoRA ablation and a post-trained 9B (Qwen3.5-9B instruct) are the next probes.
  • Answered for tonight: the 35B-A3B step buys knowledge (+7 MMLU) and dates, costs noisy-label classification and calibration, nets +1 pp. A post-trained dense 9B is the untested middle.
  • Does the unknowable-record training transfer to unseen kinds of missing evidence (scienthoon's org-rule priority is the external test)?
  • Is Kev's over-confidence on PAWS a label-noise problem (the pairs are adversarially close) or a capability one? C5 in round 3 separates them: if teacher-label disagreement predicts the confident errors, it is the former.

History

Current decision

Use Qwen3-0.6B-Base as the research baseline. Keep the released Qwen2.5 checkpoint as a historical reference, not as an equally funded development track. Preserve the MacBook training path; run the controlled studies on Modal H100s.

On the same v2 recipe and data, Qwen3's development accuracy was 81.6% / 79.3% across two seeds, versus 74.2% / 65.8% for Qwen2.5. Transfer accuracy was 62.0% / 62.1% versus 60.5% / 48.2%. This supports our backbone choice, not a claim that every Qwen3 model dominates every Qwen2.5 model.

Evidence: ablation-v2 ledger, Qwen3 seed 0, Qwen3 seed 1.

Evidence and corrections

  • Original Kev versus Jev on familiar sources: 79.7% versus 81.1% micro accuracy; macro difference −1.8 points, 95% CI [−5.5, +1.7]. An interval including zero is not evidence of equivalence. Comparison, figure.
  • Original Kev versus Jev on transfer-v1: 63.3% versus 82.3%, macro difference −19.1 points, CI [−23.1, −15.0]. The overlap check covered 1,360 decision-v1 training/calibration states, not every example used to train the released checkpoint. Public pretraining overlap is unknown for both models. Comparison, manifest.
  • Lower aggregate ECE on transfer-v1 did not establish generally better calibration. Qwen3 trial 6 gets 50% authorization accuracy with 99.6% mean confidence on transfer-v2. Its familiar-source ECE falls from .073 to .026 after fitting temperature on calibration; this does not establish transfer calibration. Result.
  • Jev's zero observed argmax flips do not prove architectural invariance. Its probabilities move under permutation. Jev transfer-v1 report.
  • A none-present accuracy of 25% means 75% wrong, not necessarily 75% selecting none. Report actual none-option mass and selection separately. Original comparison diagnostics.
  • MMLU and domain-transfer failures do not by themselves identify a knowledge versus readout bottleneck. We need a controlled capacity/data experiment.
  • Two v2 issues were found: siblings shuffled their sentence order independently, and calibration took the first slice of a family-ordered synthetic list. Pair metrics also compared option indices rather than semantic keys. Fix these under v3, with tests; do not rewrite v2 again.

Nimble research

Bespoke Nimble, model card, dataset guide.

The inspected release trained a Qwen3.5-9B LoRA adapter on 2,676 synthetic contrastive records. It reported 90.1% agreement with synthetic reference labels versus Jev's 93.2% on 324 records from six source families. These are useful external research results, not measurements on our evaluation suite or proof of broad parity. In the inspected checkout, data/ was ignored and the dataset guide required separately supplied files; no directly usable dataset was found during that review.

Adopt minimal factual edits, executable labels where possible, evidence checks, and grouped splits. A check on structured fact dictionaries does not independently validate the English rendering. Separate model calls also do not eliminate correlated synthetic-label errors. Hard outcome labels with a proper scoring rule are sufficient; teacher probabilities are not required.

Nimble scores letter tokens and supports a hybrid recurrent backbone through separate field execution. Our current packed mask alone does not isolate recurrent state, so Qwen3.5 is not a drop-in replacement. This does not make hybrid models fundamentally unusable.

v3 protocol (approved)

1. Correctness and immutable artifacts

  • Preserve all existing suite versions and run directories. New freezes and Modal trials refuse to overwrite existing paths, including failure artifacts.
  • Match sentence order and option order inside a minimal pair. Change one decisive fact only.
  • Stratify calibration by family while keeping pairs/groups together.
  • Reject incomplete or duplicated pairs. Compare semantic answer keys, not their positions.
  • Preserve locked-test bytes. New held-out structures/renderings are appended only to a new version. Do not score the locked test during this study.
  • Verify exact semantic-state separation among newly generated groups. Do not claim fuzzy or pretraining decontamination. Inherited legacy test overlap with newly generated controls is not certified by this audit.

2. Compositional policy data

Generate an explicit rule tree, facts, executable reference label, and a textual rendering. Include numeric comparisons (<, ≤, >, ≥, =, inclusive ranges), entity equality, elapsed-date comparisons, AND, OR, NOT, exceptions, and conditional precedence.

For each situation create both:

  1. Relevant intervention: change one fact and require the answer to change.
  2. Irrelevant intervention: change a routing reference and require the answer to stay unchanged.

Keep all four records in one split and one bootstrap unit. Require removal of the decisive fact to make the relevant decision unknown in the executable rule. Test truth tables and numeric/date boundaries, then manually inspect rendered samples.

Eight rule structures are trainable; three new compositions are transfer-development only; three further structures and a reserved rendering style are locked-test only. Authorization and deadline template families remain excluded from training, although related logical primitives are intentionally trained. This measures compositional/template transfer, not unseen primitive knowledge.

3. Matched data-versus-capacity experiment

Backbone Corrected legacy policy data Compositional policy data
Qwen3-0.6B-Base Control Data effect
Qwen3-4B-Base Capacity effect Combined effect
  • Identical 3,000 public training records from ten permitted sources in both arms.
  • Exactly 448 synthetic training records per arm, replacing rather than adding examples.
  • Shared, family-stratified calibration and shared development/transfer sets.
  • Two epochs, LoRA rank 16, identical learning rate per comparison, effective batch size eight. Memory-saving microbatching must preserve per-record loss weights, including the last partial batch.
  • Start with one seed per cell; repeat the cells with a second seed if smoke validation and the budget permit. Do not select and report only the better seed.
  • Record total forward tokens, steps, memory, training/evaluation wall time, full resolved configuration, dependency lock hash, source hashes, and base revisions. Equal record exposure is not equal FLOPs.

4. Selection and uncertainty

Primary score remains macro development NLL, with separate familiar-task retention and transfer reporting. Also report Brier, raw/calibrated ECE, confident-error rate at probability ≥0.9, accuracy at 50%/80% coverage (including ties), relevant-pair both-correct rate, and irrelevant-pair invariance/both-correct rates.

Temperature is fit on calibration only and applied unchanged to transfer. Neither the agent nor a candidate can adjust the evaluator through the trial config.

Research screening includes complete coverage, isolation, original-task retention, transfer accuracy/Brier non-regression, at least 70% held-out pair correctness, and at most 10% confidently wrong transfer answers. These are predeclared provisional thresholds, not production guarantees. A development win can become a candidate for a locked test; it must never automatically become a released model.

5. Compute and spending

No local training job needs to be interrupted. The Modal workspace currently reports $9.46 metered usage before this phase (covered by credits). Verified H100 rate: $3.95/GPU-hour, plus CPU/memory/storage. Modal pricing.

Use at most four concurrent containers, no automatic trial retries, and an 1,800-second per-trial timeout. Launch-time cost admission uses the GPU rate plus bounded CPU/memory requests; it excludes image build/startup/storage and is not an account-level hard spending cap. Keep the phase within the previously discussed $100 total budget, with an initial study compute bound below $20. Preserve failures rather than silently retraining or overwriting them.

Modal documentation: images, volumes, GPU, secrets.

Overnight autoresearch (branch research/overnight-1, PR #3)

Authorized: up to $500 of Modal credits; spend baseline $27.17 at 19:45. Rules unchanged: selection on development partitions, locked test read at most once per promoted candidate, no evaluator changes from a trial config, every trial through kev.experiment.execute_trial with provenance. New tonight:

  • kev/autoresearch.py: bounded hill-climb over the allowlisted config space; leaderboard across all studies (runs/leaderboard.md); spend ledger from modal billing; auto-maintained log below.
  • Architecture switches (flags, default off): option_isolation (option spans are isolated sub-branches with shared positions; permutation invariance exact by construction, 1.2e-7 measured on the real model), special_embeddings (train the five delimiter embeddings), head_dim.
  • Suites: decision-v4 (10k public + 448/arm synthetic) and decision-v5 (20k public + 1,792/arm), both with dev/test bytes identical to v3 so every number since the matched study is comparable. transfer-v4/v5 are byte-identical to v3.
  • Reference: Jev on decision-v4 dev acc 0.845, Brier 0.237, trained-structure pairs 0.86; on transfer-v4 dev acc 0.857.

Sequence: 0.6B screening rounds (cheap, one seed, replicate winners) -> promote the winning knobs to 4B on v4 -> 4B on v5 -> 8B once with the best recipe -> one locked-test read for the best gated candidate -> research-preview cards.

Blocked at 20:55: the Modal workspace hit its spend limit (Workspace ... has exceeded its spend limit; metered $48.50, $30 credits applied, $18.50 billed). Twelve running containers were killed (arch screen 4/8 evaluated, both 4B mix studies mid-training). Raising the limit needs the dashboard (workspace Settings -> Billing -> spend limit); the CLI cannot. Until then: evaluations of the six salvaged arch-screen checkpoints run on the MBP GPU (kev.experiment --resume), the 0.6B research preview is prepared locally, and 4B/8B work waits.

Findings so far tonight (v4 suites; Jev dev 0.845 / transfer 0.857):

  • 0.6B is saturated on transfer at 0.59-0.61 regardless of knobs (round 1: eight mutations, all within noise; lr 5e-4 and lora 64 + ord_w hurt dev). Run-to-run noise at the same seed on GPU is ~1 pp.

  • More public data hurts 4B transfer: 4B on v4 (10.9k) transfer 0.704 / 0.735 vs 0.747 / 0.750 on v3 (3.4k), while dev rises 0.825 -> 0.849. The lever at 4B is the data mix, not volume; synthetic_repeat / public_frac knobs added; the v4-vs-v5 mix studies were killed before finishing.

  • Option isolation trains normally at 0.6B (dev 0.800) with a measured flip rate of exactly 0.0; transfer 0.58-0.60 (parity with the plain encoding; six salvaged arch-screen trials all within noise of the incumbent).

  • Fine-tuning loses base capability (scripts/base_mmlu_probe.py: the base model, zero-shot, next-token logits over option letters, same 80 frozen items per task):

    task (transfer-v4 dev) 0.6B base Kev 0.6B 4B base Kev 4B 8B base Kev 8B Jev
    MMLU 0.425 0.46 0.688 0.60-0.66 0.762 0.65 0.90
    PAWS 0.70 0.56 0.787 0.56-0.71 0.85 0.70-0.78 0.79
    SciQ 0.912 0.86 0.975 0.95-0.97 1.0 0.99-1.0 0.99
    QNLI 0.775 0.85 0.812 0.84-0.91 0.887 0.84-0.88 0.925
    Emotion 0.287 0.50 - 0.53-0.62 0.30 0.53-0.56 0.60
    TweetEval offensive 0.588 0.69 - 0.64-0.80 0.637 0.69-0.75 0.81

    The fine-tune helps on classification-shaped tasks and hurts on knowledge and paraphrase: -11 pp MMLU at 8B, -14 pp PAWS at 0.6B. The 8B base zero-shot already beats Jev on PAWS. Hypotheses under test: low-drift LoRA (lowdrift-4b-v4: fewer target modules, r=4-8, lr 5e-5..1e-4, 1 epoch) and a knowledge-MCQ training mix (knowledge-4b-v6: ARC-Challenge, OpenBookQA, CommonsenseQA added as trainable sources; MMLU/SciQ stay eval-only).

  • 4B on v5 (20k public, compositional arm, 1 epoch): dev 0.858 (best 4B dev), transfer 0.713: more public data keeps raising in-distribution accuracy and not transfer.

  • The learning rate is the lever at 4B and 8B. Default lr 2e-4 erodes base capability; lr 5e-5 (everything else equal) gives transfer 0.759 / 0.758 at two seeds on v4 and 0.755 / 0.761 on v6, +4.7 pp [+0.4, +9.6] vs the default, with the best Brier (0.346). Fewer LoRA target modules and smaller ranks help less; combining low lr with public_frac / synthetic_repeat / option isolation does not stack (0.748-0.752). Knowledge MCQ sources (v6) lift in-distribution accuracy to 0.86 and MMLU to 0.70 without moving transfer. 8B at lr 5e-5: dev 0.869, transfer 0.774, Brier 0.339 (best of any size) but +0.4 pp [-3.9, +4.5] vs 4B on transfer. The 4B->8B step is flat for this recipe; the remaining gap to Jev (0.857) is MMLU (0.69 vs 0.90; the 8B base gets 0.76 zero-shot), PAWS, emotion.

  • Mix results at 4B: public_frac 0.33 (+synthetic_repeat 2 + isolation) 0.752 at lr 2e-4; synthetic_repeat 3 hurts (0.686, PAWS 0.45); 1 epoch is better calibrated (Brier 0.379, confident errors 2.7%) at equal accuracy; the 2-epoch 4B run on v5 (23.6k records, 5.9k steps at lr 2e-4) collapsed (dev 0.58) - long runs at lr 2e-4 are unstable.

  • Research previews published, each with one exploratory (ungated) locked-test read, all above their development numbers: Kev-0.6B dev 0.805/0.598 -> test 0.819/0.631; Kev-4B (lowdrift-4b-v4/01-trial-1) dev 0.843/0.759 -> test 0.852/0.794; Kev-8B (recipe-8b-r1/00-trial-0) dev 0.869/0.774 -> test 0.869/0.799. None passes the predeclared 70% held-out-pair screen (0.11 / 0.62 / 0.61), so none is a versioned release.

  • Kev-0.6B research preview published (jaredpalmer/kev-0.6b, card docs/model-cards/kev-0.6b.md); one exploratory (ungated) locked-test read: decision 0.819, transfer 0.631 (runs/locked/kev-06b-preview-ungated/summary.json).

Overnight synthesis (as of 02:00; spend ~$140 of $500)

What moved the needle, in order of effect size, all on the same frozen development items:

  1. Backbone capacity 0.6B -> 4B: +14-19 pp transfer (matched data). 4B -> 8B: +1.5-2 pp at the low-lr recipe (4B 0.759/0.758, 8B 0.774/0.779 at two seeds each).
  2. Learning rate 2e-4 -> 5e-5: +4.7 pp [+0.4, +9.6] at 4B, replicated at two seeds and two suites; best Brier; the mechanism is reduced drift from the base model (base zero-shot probe). 3e-5 and 2e-5 are equivalent to 5e-5.
  3. None-of-the-above minimal pairs: none_present accuracy 0.75 -> 0.78-0.85 (0.6B), 0.85-0.93 (4B).
  4. Compositional policy data: +4-5 pp at 4B on v3 (CI touching zero), nothing at 0.6B; teaches trained structures (0.85-1.0) and transfers partially to unseen ones (0.5-0.67 at 4B/8B).

What did not work: more public data (raises dev, lowers or flattens transfer; a 2-epoch 23.6k-record 4B run at lr 2e-4 collapsed); synthetic oversampling x3 (hurts PAWS badly); LoRA rank/target ablations (within noise once lr is low); option isolation (exact permutation invariance at no accuracy cost, but no accuracy gain); special embeddings; head_dim; perm_kl; ord_w; 3 epochs; knowledge MCQ sources (dev +2 pp, MMLU +2-5 pp, transfer flat). Fourteen one-knob mutations around the low-lr incumbent at 4B all landed in 0.748-0.767: the config space is exhausted for this data and evaluation; run-to-run noise at a fixed seed is ~1 pp.

Also tried after the synthesis: WiSE-FT-style interpolation toward the base at inference (KEV_LORA_SCALE): alpha 0.75 0.765 (noise), alpha 0.5 0.736 (-4.4 pp, CI excludes zero) - the head depends on the fine-tuned features, so weight-space interpolation is not a lever. Round auto-4b-r1 (head_lr, weight decay, rank 8, head_dim 1024, perm_kl, synthetic x2): all 0.755-0.767; option isolation at low lr 0.729 (-5.8 pp, significant) - isolation costs accuracy at 4B.

Where the remaining gap to Jev (0.857 transfer) lives, per task at 8B: MMLU 0.69-0.74 vs 0.90 (the 8B base is 0.76 zero-shot - our readout still loses knowledge), PAWS 0.75 vs 0.79 (base 0.85), Emotion 0.55 vs 0.60, TweetEval 0.71 vs 0.81, deadline 0.55-0.70 vs 0.925. Held-out policy pairs 0.61-0.67 vs the 0.70 screen.

Next levers the evidence points at (not config knobs):

  • Knowledge readout: the pointer head under-uses what the base knows. Try a hybrid readout that adds the base model's own letter/option-token logits (frozen, zero-drift) to the pointer logits, or distill the base model's zero-shot distribution on knowledge-shaped questions into the head (self-anchoring, no Jev).
  • Date/ordinal reasoning: deadline stays near the middle level; needs either scratchpad-free arithmetic data with varied surface forms or a Score readout that models cumulative levels directly.
  • Held-out structure generalization: more rule structures (not more pairs per structure) and rendering styles.
  • Evaluation: the 70% pair screen is within reach at 8B (0.59 / 0.67 / 0.61 across three seeds) but not met by any seed, so no gated locked-test read happened tonight; the published previews carry ungated reads.

Final replication (03:30): 4B lr 5e-5 at three seeds transfer 0.759 / 0.758 / 0.759; 8B lr 5e-5 at three seeds 0.774 / 0.779 / 0.774. Both recipes are stable to ~0.5 pp. Spend for the night: ~$180 of the $500 authorized (89 trials indexed; runs/leaderboard.md).

Toward v0.2: crossing the release screen (2026-09-19)

Where the 8B loses the 70% held-out-pair screen (64 relevant pairs; needs 45, has 39-43; Jev 55):

held-out family Kev-8B seeds 0 / 1 Jev missing from training
authorization 20/20, 20/20 20/20 -
(A or B) and C 6/8, 8/8 6/8 -
if A then not B else C 2/8, 4/8 5/8 negation nested inside a conditional
(A and B) or not C 1/8, 2/8 7/8 negation nested inside a disjunction
deadline (3-level Score, grace period) 10/20, 9/20 17/20 any ordinal-threshold Score family; date arithmetic only as yes/no

Training had eight fixed rule shapes with negation only at the top level, and no Score question whose levels are threshold intervals. The 8B base scores 0.58 zero-shot on the held-out rule items (near chance), so rule composition is learned from our synthetic data, which is why structural coverage is the lever.

decision-v7 (manifest; dev/test bytes identical to v4, so every number is comparable; scored against transfer-v4): 10k public (v4 pool) + legacy policy arm with four new ordinal Score families (warranty claim by date, SLA response hours, late-fee days, volume discount; 896 records) + compositional arm from 60 random rule trees with negation anywhere, depth 2-3, rendering styles 0/1/3/4 (1,680 records). Held-out and locked structures are excluded by a canonical key that is invariant to commutation, leaf numbering and De Morgan pushing (kev/composition.py canonical, push_negation, sample_trees).

Release rule tightened: the screen must pass on every seed of a config (kev.autoresearch release-check), not one seed of 64 pairs. Study v7-release-candidates: 4B and 8B at lr 5e-5, two seeds each.

Results so far (v7 and the untrained baselines)

Untrained baselines on the same frozen items (scripts/base_mmlu_probe.py, zero-shot letter logits, Modal):

transfer-v4 dev 8B base, untrained 30B-A3B base, untrained Kev-8B Jev
accuracy 0.726 0.707 0.774 0.857
Brier 0.366 0.365 0.339 0.211
MMLU 0.75 0.79 0.69 0.90
PAWS 0.84 0.82 0.76 0.79
Emotion 0.30 0.31 0.57 0.59
held-out rule pairs 0.55 0.44 0.61 0.86

Kev-8B vs its own untrained base: +5.8 pp [+1.8, +10.0]; vs the untrained 30B-A3B: +9.7 pp [+4.7, +14.4]. The recipe does real work (rule composition, classification-shaped tasks) and loses on knowledge/paraphrase, where both untrained models beat it (MMLU 0.75-0.79 vs 0.69; PAWS 0.82-0.84 vs 0.76). "A bigger untrained MoE gets Jev-class results for free" is false on this suite; "the fine-tune erodes base capability" is confirmed a third way.

decision-v7 at 4B (v7-rc3): transfer 0.773 / 0.790 (v4 recipe: 0.759 x3), dev 0.858 / 0.854, held-out pairs 0.62 / 0.73. The random-structure data moves the two failing rule families: (A and B) or not C 0.66 -> 0.75 / 0.97, if-then-not 0.62 -> 0.75 / 0.88. The ordinal Score families did not move deadline (0.60 / 0.53). Seed 1 clears the 70% screen; seed 0 does not, so under the two-seed rule this is not yet a release candidate. 8B on v7 and a third 4B seed are running; the deadline family needs a different idea (the model still hedges to the middle level).

Anchoring (kev.anchors + --anchor_w, KL toward the frozen base's zero-shot distribution, targets keyed by option key; anchor-4b-v6): w=0.5 on the knowledge MCQ sources transfer 0.773 (+2.6 pp [-1.8, +6.8] vs the un-anchored v6 recipe), w=0.3 on everything 0.761 with the best calibration of any 4B (Brier 0.338, confident errors 3.4%), w=1.0 0.741. MMLU stays 0.66-0.69 and PAWS 0.68-0.70 under every setting. Not a lever: at lr 5e-5 the 4B already matches its base on MMLU (0.69 vs 0.688); the remaining PAWS gap (0.70 vs base 0.79) is not closed by anchoring the output distribution, which points at the readout/format rather than at drift (the third outcome in the anchoring experiment's table).

Final v7/v8 read (21:20). 4B on v7, three seeds: transfer 0.773 / 0.790 / 0.770, held-out pairs 0.62 / 0.73 / 0.67. 8B on v7, two seeds: transfer 0.796 / 0.774, pairs 0.69 / 0.64. 4B on v8 (v7 + shipping_delay, a day-precision date-threshold Score family built for the deadline skill): 0.777 / 0.759, pairs 0.69 / 0.56, deadline 0.53 / 0.45 - the extra family did nothing. No config passes the screen on every seed; 8B seed 0 misses by one pair (44/64). The deciding family is deadline (0.45-0.60 for every Kev; untrained 8B/30B bases 0.53-0.55; Jev 0.93): day-precision date arithmetic to a 3-level ordinal in one forward pass looks like a capability limit at <= 8B for this readout, not a data gap - two purpose-built training families did not move it.

Published as updated research previews (no version tag; each card records one locked read): Kev-4B = v7-rc3/01-trial-1 (dev 0.854 / 0.790; locked 0.856 / 0.806), Kev-8B = v7-final/00-trial-0 (dev 0.863 / 0.796; locked 0.870 / 0.780). The 8B locked read was interrupted once before any aggregate existed and redone (recorded in its summary).

What would cross the bar, in order of my confidence: (1) an ordinal readout that models cumulative thresholds for Score questions (the models hedge to the middle level on deadline); (2) explicit day-count rendering in training states (teaching the arithmetic is not the point of a decision model; giving it the number is); (3) capacity beyond 8B. Spend to date ~$250 of $500.

Release (2026-09-20). The three checkpoints are published as the Kev family, no version tags (nothing overlapping to version; the 0.5B prototype stays on the Hub for reference). The 70% held-out-pair screen is no longer a publication gate (it was within seed noise and decided by one arithmetic family); it stays in kev.experiment as a research gate and in release-check. Final selection, every checkpoint the best of its size on the leaderboard: Kev-0.6B = v7-06b/02-trial-2 (dev 0.801 / 0.620, locked 0.808 / 0.642; three v7 seeds 0.613 / 0.605 / 0.620, all above the previous 0.598), Kev-4B = v7-rc3/01-trial-1, Kev-8B = v7-final/00-trial-0. Serving: fp32-merged LoRA, SDPA on MPS, state-prefix KV cache, shape bucketing (all parity-tested).

Qwen3.5 generation (2026-09-20). Same data and recipe on Qwen3.5-4B/9B bases (hybrid Gated DeltaNet + attention; questions run as causal rows from a shared state instead of under a packed mask). Development criteria set in advance were not met (accuracy CI includes zero; deadline 0.72 not 0.75); the single locked-test read shows Kev-9B 0.837 vs Kev-8B 0.780 out of domain (+7.3 pp [+2.8, +11.7], Brier 0.243 vs 0.327) and Kev-4B 0.832 vs 0.806. Published as the current family: jaredpalmer/kev-9b, jaredpalmer/kev-4b (Qwen3 weights at tag qwen3) and jaredpalmer/kev-0.8b (locked 0.827 / 0.668 vs Kev-0.6B 0.808 / 0.642, +4.8 pp [+0.2, +9.3]); the Qwen3 checkpoints stay published as the fast Mac option and are no longer developed. The deadline family is a training-data problem, not a readout problem (issue #8; adapter-merge probe). Full log: Qwen3.5 port under History.

Ops: two studies were lost to the local client disconnecting (Modal cancels .starmap inputs when the caller dies; --detach keeps only the last-triggered function). Studies now fan out server-side from the deployed app (modal deploy modal_app.py; run_study spawned, pull --name afterwards). ~$35 of GPU time was lost to this.

Qwen3.5 port (2026-09-20)

Moved here from PLAN_Qwen35.md (merged 2026-09-21; section numbers §1-§10 below are the ones the model cards and scripts/compare_q35.py cite).

Status: completed 2026-09-20. The plan below was executed in full; §10 records the results. The Qwen3.5 family (Kev-0.8B / 4B / 9B) shipped the same day.

Status: proposal for review, 2026-09-20. Nothing here has been started except the probe in section 2. Numbers are on the same frozen items as every other number in this repository (evals/v4/transfer-v4, development partition, 764 records); untrained bases are read zero-shot from next-token letter logits with scripts/base_mmlu_probe.py, the same probe used for every untrained row in the README.

1. What "latest Qwen base" means, checked

Every Qwen release after Qwen3 shares one text architecture, qwen3_5_text, and only Qwen3.5 has Base checkpoints (Hub listing, author=Qwen):

family released Base weights architecture (from config.json)
Qwen3 2025 0.6B, 1.7B, 4B, 8B, 14B, 32B (Qwen3-8B-Base) dense attention, every layer
Qwen3.5 Feb–Mar 2026 0.8B, 2B, 4B, 9B, 35B-A3B (Qwen3.5-9B-Base, Qwen3.5-4B-Base) hybrid: 32 text layers, 24 Gated DeltaNet ("linear_attention") + 8 full attention, full_attention_interval: 4 (9B config)
Qwen3.6 Apr 2026 none (27B, 35B-A3B, post-trained only) qwen3_5_text, 64 layers, 16 attention / 48 DeltaNet
Qwen3.8 Aug 2026 none (27B, 2.4T-A95B, post-trained only) qwen3_5_text, same shape as 3.6-27B

So: Qwen3.5 Base is the newest generation Kev can train on, and building for its architecture builds for Qwen3.6/3.8 the day their Base weights appear. Model details for the two sizes we would use (read from the configs):

Qwen3.5-4B-Base Qwen3.5-9B-Base
Hub revision (pin) 1001bb4d826a52d1f399e183466143f4da7b741b 68c46c4b3498877f3ef123c856ecfde50c39f404
text layers 32 (8 attention, 24 DeltaNet) 32 (8 attention, 24 DeltaNet)
hidden 2560 4096
attention heads / KV heads 16 / 4 16 / 4
DeltaNet value heads 32 32
vocab 248,320 248,320
context 262,144 262,144
checkpoint parameters (incl. vision tower) 4.7B 9.7B

The checkpoints are Qwen3_5ForConditionalGeneration (a vision tower is bundled). Loading through AutoModelForCausalLM gives the text-only Qwen3_5ForCausalLM with .model = Qwen3_5TextModel; verified on Qwen3.5-0.8B-Base under transformers 5.17.0 (forward on CPU, top token for "The capital of France is" → " Paris"). Our five delimiter tokens (kev/model.py SPECIAL) exist in the Qwen3.5 tokenizer with ids 248049–248062, so the encoding scheme carries over unchanged.

2. The probe: is the new base actually better for Kev?

Run 2026-09-20 on Modal H100s with modal_probe35.py (since folded into modal_app.py::base_probe; own image at the time: transformers 5.17, flash-linear-attention for the DeltaNet kernel). Reports and per-item rows: runs/probes/. Paired comparisons are record-clustered bootstraps (kev.metrics.paired_bootstrap).

untrained base, zero-shot acc Brier MMLU PAWS QNLI SciQ TweetEval Emotion authorization deadline rule: (A∨B)∧C rule: (A∧B)∨¬C rule: if-then-not
Qwen3-0.6B 0.567 0.505 0.42 0.71 0.76 0.93 0.57 0.29 0.50 0.28 0.38 0.62 0.44
Qwen3.5-0.8B 0.566 0.510 0.47 0.66 0.72 0.90 0.62 0.21 0.55 0.28 0.50 0.56 0.50
Qwen3-4B 0.671 0.392 0.68 0.79 0.78 0.97 0.65 0.29 1.00 0.55 0.56 0.41 0.47
Qwen3.5-4B 0.692 0.395 0.69 0.84 0.84 0.99 0.65 0.28 0.95 0.68 0.53 0.44 0.50
Qwen3-8B 0.726 0.366 0.75 0.84 0.89 1.00 0.68 0.30 1.00 0.53 0.59 0.62 0.62
Qwen3.5-9B 0.729 0.356 0.74 0.84 0.90 0.99 0.69 0.31 1.00 0.82 0.59 0.47 0.44
Kev-8B (trained, Qwen3-8B) 0.796 0.337 0.70 0.78 0.91 1.00 0.79 0.56 1.00 0.60 0.97 0.91 0.59
Jev 0.857 0.211 0.90 0.79 0.93 0.99 0.81 0.59 1.00 0.93 0.91 0.97 0.78

Paired deltas, new base minus old base at matched size:

  • Qwen3.5-0.8B vs Qwen3-0.6B: +0.8 pp [−3.3, +4.8]
  • Qwen3.5-4B vs Qwen3-4B: +2.1 pp [−1.5, +5.8]
  • Qwen3.5-9B vs Qwen3-8B: −0.3 pp [−4.9, +3.9]

Sources: runs/probes/qwen35-9b-base-base-transfer-v4/report.json, qwen35-4b, qwen35-2b, qwen35-08b, qwen3-8b, qwen3-4b, qwen3-06b; Kev-8B from runs/v7-final/00-trial-0/result.json; Jev from runs/jev-transfer-v4/report.json.

Reading

  1. Overall, the new generation is not a better base on our suite. At 9B vs 8B the difference is zero within noise; at 4B it is +2 pp with a CI that includes zero. MMLU and PAWS, the two places Kev loses most to Jev, are unchanged (0.74 vs 0.75, 0.84 vs 0.84). A version bump alone would not have been worth a port.
  2. But the gain is concentrated exactly where Kev is stuck. On deadline (day-precision date arithmetic to a 3-level ordinal), Qwen3.5-9B gets 33/40 items zero-shot where Qwen3-8B gets 21/40, and Qwen3.5-4B gets 0.68 where Qwen3-4B gets 0.55. This is the one family that no training data moved — two purpose-built families in decision-v7/v8 left it at 0.45–0.60 for every Kev ("Final v7/v8 read") — and it was the deciding family for the predeclared held-out-pair screen, which Kev-8B missed by one pair (0.69 vs 0.70).
  3. The rule-composition columns are irrelevant to the choice of base. Every base is near chance on them zero-shot; Kev learns them from the synthetic data (Kev-8B 0.91–0.97). What carries over from the base is knowledge and arithmetic, and the low-learning-rate recipe was found precisely because it preserves base capability ("The learning rate is the lever").

Verdict: proceed, on the narrow hypothesis that a Kev trained on Qwen3.5-9B keeps most of the base's 0.82 on deadline, which would lift held-out-pair correctness from 0.69 to roughly 0.80 and put the overall out-of-domain number at ~0.81 (Jev 0.857). The probe does not justify expecting gains on knowledge or paraphrase. Everything below is scoped to test that hypothesis at controlled cost, with the existing Kev-8B as the paired baseline on the same items.

3. Why the port is real work: the hybrid breaks the packed mask

Kev's forward pass today packs the state and every question into one sequence and uses a block-causal additive mask so that a question sees the state but never another question (encode, branch_mask_batch, hidden_batch). The mask is applied inside attention. In Qwen3.5, 24 of 32 layers are Gated DeltaNet (Yang et al., 2024; implementation Qwen3_5GatedDeltaNet in transformers): a recurrent state updated token by token, plus a short causal convolution. Neither respects a per-token attention mask, so in a packed sequence question 2's tokens would read a recurrent state that has already absorbed question 1's tokens. Isolation, the property the README verifies to 4e-6, would fail by construction. transformers confirms this: in Qwen3_5TextModel.forward the mask dict is only consulted for full_attention layers (attention_mask=causal_mask_mapping[self.config.layer_types[i]]); the DeltaNet path only uses the 2D mask to zero padding.

The design: one state pass, then questions as a batch of rows

Replace "one row, masked" with "state once, branches as separate rows continuing from the state":

  1. State pass. Run the state tokens (<state> …, positions 0..Ls-1) once with use_cache=True. The cache holds the KV of the 8 attention layers and, for the 24 DeltaNet layers, the recurrent state and the conv state (cache_params.layers[i].recurrent_states, conv_states in the transformers implementation).
  2. Branch pass. Build a batch of Q rows, one per question: <q> instr <opt> o </opt> … <decide>, right-padded, positions continuing from Ls exactly as encode already assigns them. Replicate the cache along the batch dimension and run the rows. The chunked prefill path takes initial_state=recurrent_state when a cache with previous state is present (torch_chunk_gated_delta_rule(..., initial_state=...)), so a multi-token continuation from a cached state is supported by the library, not something we would hack in. This exact pattern is already running on Qwen3.5-4B in the wild: SemIf's shared.py prefills the state once, replicates the cache with cache.reorder_cache(torch.zeros(Q)), and scores right-padded suffixes in one batch (positions continuing from the prefix length, logits_to_keep at each row's last real token); its MLX backend does the same with MLX-LM's prompt cache (entry.merge([entry] * Q), prepare(lengths, right_padding)). We adopt the reorder_cache idiom rather than batch_repeat_interleave, since it is what has been exercised on this architecture. They also quantified the cost: bf16 reuse changed 5–6 of 777 argmaxes versus fresh scoring (results), the same magnitude as the bf16 noise we measured on our own prefix cache, so the fp32 parity tests stay.
  3. Readout. Gather </opt> and <decide> hidden states per row and apply the unchanged PointerHead — _readout already takes per-question index lists.

Properties: isolation is exact by construction (rows are independent tensors; no mask to get wrong), on any architecture; the state is computed once, as today; FLOPs equal the packed form (the packed sequence also processes every branch token once); the serving prefix cache we shipped (prefix, probs_with_prefix) is literally step 1 + step 2 and becomes the only path rather than an optimization. What we lose: the single-row form and the option_isolation flag (which cost −5.8 pp at 4B anyway, PLAN.md).

Training through a cache is the one uncertain mechanical step (cache tensors may be updated in place or detached). Fallback that is exactly correct and simple: for training, build rows as [state + branch_q] with the state duplicated per row. Cost is Q× the state tokens instead of 1×; our training records average ~1.3 questions, so ≤ 1.5× the state compute, and the state is short (≤ 384 tokens). Start there; optimize to the cache-continuation form only if it is measurably needed.

Equivalence check that gates everything: on the existing Qwen3-based Kev-4B, the row-batched forward must reproduce the packed-mask forward's probabilities to fp32 noise on the 24-record parity set used for the serving changes. If it does, the same code path is the reference for the hybrid.

4. Dependencies and their risks

item today needed evidence / risk
transformers >=4.51,<4.58 (pyproject) >=5.17 (qwen3_5 is not in 4.x) 5.17.0 loads Qwen3.5-0.8B-Base and runs a forward (verified in a scratch venv). Risk: 5.x API changes for the existing Qwen3 path; gate with the parity tests in tests/test_model.py (merged-vs-unmerged, prefix-vs-full, bucket padding) and a bf16 dev-set re-score of the published Kev-4B (must match runs/v7-rc3/01-trial-1/result.json within bf16 noise).
peft 0.21.0 same — peft 0.21.0 is already verified on Qwen3.5 + transformers 5 by pngwn/system-one-qwen3.5-4b-scorer-v2b (peft_version: 0.21.0, LoRA r=16 α=32, 30.5M trainable params, lr 1e-4, seq 384, 11.5k steps) LoRA target names on the DeltaNet layers, verified against Qwen3_5TextModel in transformers 5.17: in_proj_qkv, in_proj_z, in_proj_a, in_proj_b, out_proj; attention (q/k/v/o_proj) and MLP (gate/up/down_proj) names are unchanged. pngwn targets in_proj_qkv, in_proj_z, out_proj + attention + MLP. That scorer and Bespoke-Nimble-9B both score one sequence per question (pngwn: one per option, via AutoModelForSequenceClassification); neither shares the state across questions, which is what section 3 adds.
DeltaNet kernel – flash-linear-attention + triton on CUDA Without it transformers falls back to a reference PyTorch implementation ("correct but much slower", its own warning). CUDA-only. Mac serving runs the fallback; speed unmeasured — measured in phase 1 before committing to a Mac story.
Modal image pins uv.lock a second image (or the upgraded lock) modal_probe35.py (later folded into modal_app.py) already builds the transformers-5 image; the main app follows once the lock is upgraded.
suites base revisions pinned per suite (kev.suite.freeze, validated_trial requires a 40-hex base_revision for unpinned bases) pass base_revision per trial (already supported) or freeze decision-v9 = v7 with Qwen3.5 revisions pinned Dev/test bytes stay identical to v4 either way, so every number remains comparable.
vision tower – ignored AutoModelForCausalLM loads the text model only (verified). Weight download is the full 9.7B checkpoint.
vocabulary 152k 248k Embedding matrix is frozen; the pointer head reads hidden states, not logits. No change.

5. Work plan

Each phase has a stop condition. Costs are Modal H100 at the rates we have been paying (Kev-4B trial ≈ $3.50, Kev-8B ≈ $6; "Compute and spending").

Phase 0 — MMLU-Pro, a buried-state variant, and cross-benchmarks with SemIf (independent of the port; ~4 h, ~$3)

  • Add mmlu_pro to kev/data.py as an eval-only source (TIGER-Lab/MMLU-Pro, Wang et al., 2024; 10 options, answer_index), rendered as a Choice with neutral keys like the other MCQ converters. Pin the dataset revision (b189ec765a…).
  • Freeze transfer-v9 = transfer-v4 items + 200 MMLU-Pro items (new version; v4 numbers stay valid, and v9 minus the new sources equals v4 byte-for-byte for comparability).
  • Score Jev, the untrained bases, and Kev-4B/8B on it. Expectation, from openjev's report on the same benchmark: Jev ~0.83, untrained one-token readouts ~0.6, Kev well below Jev — the honest number for a one-pass model on a benchmark designed for chain-of-thought. It replaces the saturated 4-way MMLU in the knowledge column of the README.
  • Add an eval-only buried-state variant to transfer-v9: the same record with the state embedded in unrelated background text (SemIf's "irrelevant context" perturbation and localjev's 2,048-word distraction condition both found this is where small models fall over; we train at ≤ 384 state tokens and have never measured it).
  • SemIf-style untrained baseline on our suite: Qwen3.5-4B instruct (their pinned revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a) with their exact prompt (core.direct_messages: system instruction + JSON {evidence, criterion, options} through the chat template, letter logits) on transfer-v4, alongside our base-model probe row. Our probe uses the base model and a plain prompt; theirs may be the stronger untrained readout and is the one people will compare against.
  • Kev on the uncontaminated ticket set from scienthoon/jev-ood-calibration (900 rule-generated tickets × 3 questions, regenerable from a seed; live Jev numbers already exist on the same items via the same AI Gateway path we use: 89.0% / 91.7% / 44.7%, overall ECE 0.107 at 4.4× the noise floor). Two findings there shape our evaluation: Jev's miscalibration flips sign by question type (choice/score overconfident, refit T ≈ 3.3; boolean underconfident, T ≈ 0.66), and on the unknowable priority question (the label is an org rule absent from the text) Jev is at chance with mean stated probability 0.74. Add to transfer-v9 an unknowable-label family from our contrastive generator (label depends on a rule the state does not contain) where the scored behaviour is low confidence — a "knows it doesn't know" column no open model reports today.
  • Coverage at a fixed error budget as a first-class metric in kev.metrics: the maximum fraction of decisions automatable at ≤ 5% empirical error, as in AbdelStark/jev-benchmarks (live Jev: 0.83 on AG News, 0.86 on Banking77, 0.00 on Emotion). It is the operational meaning of calibration and belongs in the README table. That harness is Apache-2.0 and pluggable; a PR adding Kev as a system yields third-party numbers we did not produce.
  • Kev on SemIf's fixtures: convert their committed authored144.jsonl and perturbations108.jsonl (three families: evidence interpretation, rule application, candidate selection; 2–3 options with ids and descriptions) into System One requests and score Kev-4B/8B with the labels they ship. Their reported direct-logit numbers are 0.813 balanced accuracy on the 144 and 0.723 on the 36 perturbation bases. Also run live Jev on the 144 rows (cents) — their Jev column is agreement with published outputs on a different subset, not a live read.

Phase 1 — transformers 5 branch and row-batched forward (½–1 day, ~$2)

  • Branch qwen35. Upgrade the lock to transformers 5.x / peft ≥ 0.18; run the full test suite and the bf16 re-score of Kev-4B; fix what 5.x moved.
  • Implement DecisionModel.forward_rows (state pass + row batch) alongside the packed path; add test_rows_match_packed on the smoke checkpoint (fp32, CPU, < 1e-4) and on Kev-4B (bf16, 24 records, 0 argmax flips).
  • Load Qwen3.5-0.8B-Base through DecisionModel, run the isolation and packing checks from kev.evaluate on the untrained model (they test the mechanism, not accuracy). Stop if the DeltaNet cache continuation does not reproduce a single-row forward to fp32 noise; that would mean the library's chunked-prefill-with-initial-state path is not usable and we would need to write the continuation ourselves before spending on training.
  • Measure the CPU/MPS fallback speed of the 0.8B and 4B forward. Record it; it decides whether "serves on a Mac" survives for Qwen3.5-based Kevs or waits on MLX.

Phase 2 — the controlled experiment (~1 day wall, ~$25)

Same data, same recipe, new base. Nothing else changes, so any difference is the base.

  • decision-v7 training partition, lr 5e-5, 2 epochs, LoRA r=16, p_none_pair 0.25 (the released recipe, experiments/v7-final.json); LoRA on attention + MLP projections in all layers, DeltaNet in/out projections included (one ablation without them at 4B).
  • Cells: Qwen3.5-4B × 2 seeds, Qwen3.5-9B × 2 seeds, scored on transfer-v4 (and transfer-v9 once Phase 0 lands). Paired bootstraps against Kev-4B and Kev-8B on the same items (kev.autoresearch compare).
  • Read-outs, in order of what the hypothesis predicts: deadline (≥ 0.75 expected if base skill is retained), held-out pairs (≥ 0.70 on both seeds is the old release screen), overall transfer, MMLU/PAWS retention (should be ≥ Kev-8B's 0.70/0.78; if lower, the recipe is eroding the new base more and a small lr sweep at 3e-5/2e-5 is the next $10), Brier and confident-error rate.
  • Stop if Kev-9B(3.5) is not above Kev-8B on transfer with a CI excluding zero and deadline did not move. Then the result is written up in PLAN.md as a negative and the family stays on Qwen3.

Phase 3 — promotion (only if Phase 2 passes; ~½ day, ~$5)

  • One locked-test read for each winning size (modal_app.py::locked_test), gated if the pair screen passes on both seeds.
  • Cards for kev-4b/kev-9b on Qwen3.5 (naming: the Hub repo is by size, so jaredpalmer/kev-9b; kev-8b stays as is), README table and figures regenerated from result files (scripts/plot_family.py, scripts/plot_tweet.py), the "untrained base" rows switched to Qwen3.5.
  • Serving: the row-batched path is the prefix-cache path; re-run the serving benchmark in the README's "Serving performance" section on CUDA.
  • Mac serving through MLX-LM, promoted from "deferred" on SemIf's evidence that MLX-LM runs Qwen3.5 with prompt-cache replication today (docs/MLX.md: MLX 0.32.2, MLX-LM pinned past the Qwen recurrent q/k-norm fix, vision weights stripped by the sanitizer, 4/8-bit in memory). Kev's additions are small: load the fp32-merged LoRA weights (we already merge at load) into the MLX-LM Qwen3.5 text model, take hidden states from the inner model instead of lm_head logits, and apply the pointer head in MLX. Gate with the same parity tests against the PyTorch fp32 path. Measure the torch fallback first (Phase 1); if it is within ~2× of today's MPS latency, MLX waits.

Deferred, deliberately

  • Qwen3.5-35B-A3B-Base: 70 GB in bf16, MoE, serving footprint out of the laptop story; only after the 9B result.
  • A browser/WebGPU path (SemIf's distribution advantage, via wllama/GGUF). Kev's pointer head and per-token positions do not fit llama.cpp as-is; with the row-batched design a GGUF export of the merged model plus a small JS head is conceivable, but it is its own project.
  • Post-trained Qwen3.6/3.8 as bases: changes the recipe and removes the untrained-base comparison; revisit when Base weights exist.

6. Competitive context: SemIf

SemIf (formerly OpenJev; 2.3k stars) is an untrained readout: frozen Qwen3.5-4B instruct, chat-template JSON prompt, probabilities from letter logits, with prefix-reuse and parallel-suffix modes and a WebGPU browser demo. Its own results page states the probabilities are "uncalibrated as decision confidence" and that "the next justified phase is targeted training for decision semantics and calibration" — i.e. Kev. What it means for us:

  • It validates the port design (section 3) and hands us the MLX path (Phase 3). Its author solved the hybrid-cache replication problem on exactly our target base.
  • It sharpens the claim Kev has to make. Their headline is "3.8 pp behind Jev" on agreement with Jev's published answers over 102 curated public cases. On labelled frozen items, untrained Qwen3.5-4B scores 0.692 on our suite and Kev-4B 0.790: training the readout is worth about +10 pp at 4B, and it is what buys calibration (Brier, confident-error rate), which they do not measure. Once a Qwen3.5-based Kev-4B exists the comparison is apples to apples on the same base, which is the cleanest possible demonstration of what training adds.
  • Their evaluation is thinner than ours (144 authored rows, agreement rather than ground truth, no live Jev, no paired uncertainty). Phase 0 scores both ways on frozen items and runs live Jev on their rows; the neutral-ground position is worth more than winning any single cell.
  • Their distribution is stronger than ours (browser demo, "no waitlist"). Not something this plan addresses; noted in deferred work.

Laya (convaiinnovations), reviewed 2026-09-20

Laya (code, 957 Hub likes) is the other trained open decision model with weights, and it is built the opposite way from Kev: a bidirectional encoder (ModernBERT-large, 395M, fully fine-tuned; a 322M mmBERT variant for 100+ languages) plus a 2-layer transformer head that scores each option at its own [MASK] token (common.py). Sequence per question: [CLS] type+instructions [SEP] [MASK] opt0 [MASK] opt1 … [SEP] state [SEP], 512 tokens (192 reserved for options). One sequence per question: "all questions in one forward pass" means a batch, so the state is re-encoded for every question and cost scales linearly (T4: 39.5 ms for 1 question, 158.6 ms for 10). Trained with what they call RLCD: REINFORCE with Gaussian logit noise against a strictly proper reward (log score + 0.5·spherical + ranked probability score for ordinals, group-mean baseline). In expectation that objective has the same optimum as the log loss Kev minimises directly; the RL framing adds variance, not a different target. Its Jev numbers are third-party published figures, not measured.

What it teaches us, and what it does not:

  • Encoders are strong in-distribution and near chance off it. Their own limits section: base checkpoints score 0.362 zero-shot on their typed-decisions benchmark (random 0.318, majority 0.461); the 0.766 headline is a checkpoint fine-tuned on that benchmark's training split. They report no held-out-source number at all. That is the failure mode transfer-v4 was built to expose, and the reason Kev's headline is out-of-domain accuracy on never-trained sources. Any comparison we publish should score Laya on our frozen OOD items (their SDK is pip-installable; agent.predict(state, questions) takes System One shaped questions, so the conversion is trivial).
  • High-cardinality choice is a real Kev advantage. Laya scores 0.425 on Banking77 (77 options at 3–4 tokens each in a 192-token option budget); pngwn's scorer caps options at 16; Jev supports 255. Kev-0.5B scored 0.86 on 77-way Banking77 with the pointer head and no option budget. We should measure and state it (a ≥ 50-option column in the README table).
  • Calibration as shipped is worse than ours, fixed by finer temperature grouping. Laya ships at ECE 0.466 (multilingual 0.314) and reaches 0.081 with one temperature per (question type, option count). Kev ships at 0.09 in-distribution raw; our single fitted temperature did not transfer out of domain. Their grouping is a cheap thing to test on our dev/test split (Phase 0 scope, one afternoon).
  • Multilingual is a differentiator Kev gets for free from the base and has never measured. Laya needs two checkpoints and a script-detecting router because ModernBERT is English-only. Qwen3 and Qwen3.5 are multilingual; one Kev should cover XNLI/MASSIVE languages without routing. Eval-only multilingual slices belong in transfer-v9.
  • The "act/escalate" head is a product idea, not a model idea. It is an MLP over (top-1, margin, entropy, K) features plus the [CLS] vector predicting act-vs-escalate. Kev can expose an equivalent abstain flag from the same probability features, calibrated on the development partition, with no retraining.
  • Nothing to adopt from the training objective. Log loss is already a strictly proper scoring rule; we tested the ranked probability score as an extra term (--ord_w) and it did not help; the spherical term is unlikely to differ. Their soft-target training against teacher distributions is closer to distillation; Kev deliberately trains on labels, not Jev outputs.

Where Kev sits in the field

multimodalart's Jev Reproductions Tracker lists ~60 artifacts (SemIf, Laya, Nimble, pngwn's Qwen3.5-4B scorer, several RLCD LoRAs, localjev, benchmarks). Kev is not on it. Submitting it is free and is the single cheapest distribution action available.

7. Decision criteria, stated in advance

Ship a Qwen3.5-based Kev-9B as the new top of the family only if, on the same 764 items: transfer accuracy ≥ Kev-8B's 0.796 with the paired CI excluding zero, deadline ≥ 0.75, MMLU and PAWS not below Kev-8B by more than seed noise (~1 pp), and Brier ≤ 0.34. Ship a Qwen3.5-based Kev-4B if it beats Kev-4B (0.790) by the same test. Otherwise the outcome is a documented negative result and the Qwen3 family remains the release. Locked test read once per candidate, as always.

8. Future exploration after the re-architecture

Not part of this plan's budget; ordered by expected value per dollar once a Qwen3.5-based Kev exists. Each is one controlled experiment on the frozen suites.

  1. Multilingual out-of-domain slices (XNLI in 14 languages, MASSIVE intent in 51; eval-only, in transfer-v9). Zero training cost; tests whether the multilingual base carries the trained readout across languages, which would make one Kev cover what Laya needs a router and two checkpoints for. ~$2.
  2. Finer calibration — one temperature per (question type, option count), fitted on the development partition, evaluated on the locked test only once at promotion time. Two independent sources now point here: Laya (ECE 0.466 → 0.081 with this grouping) and scienthoon's Jev study (sign of miscalibration differs by type on the same inputs). Ours says a single temperature does not transfer. ~$0.
  3. A high-cardinality column in the README table (Banking77 77-way held out, plus a synthetic 100–255-option family), where Kev's pointer head has no option budget and every open competitor caps or collapses. ~$3.
  4. An abstain signal — a calibrated "uncertain" flag from (top-1, margin, entropy, K), reported alongside probabilities in /v1/systemone; product-level, no retraining. Evaluate as selective accuracy at fixed coverage on the OOD suite.
  5. Small-model path via an encoder: ModernBERT-large / mmBERT with Kev's pointer head and a block mask that keeps state tokens from attending to questions (so the state encoding is question-independent and shareable). Answers, for ~$5, whether a 400M encoder can beat Kev-0.6B's 0.62 out of domain or whether Laya's near-chance-off-distribution result is intrinsic to encoders. If it wins, it replaces Kev-0.6B for the 30 ms latency tier.
  6. Self-distillation for the small model: Kev-9B's probabilities as soft targets for Kev-0.6B/2B on the same training partition (no Jev outputs involved). Laya's soft-target training and Nimble's teacher setup both suggest gains at small sizes; our rule against training on Jev outputs is untouched.
  7. Ordinal readout for Score — the fix proposed in PLAN.md for the deadline family; if the Qwen3.5 base closes most of the gap by itself (section 2), this drops in priority, which is the cheapest possible outcome.
  8. Qwen3.5-35B-A3B-Base and, when Base weights ship, Qwen3.6/3.8 — same code path, no new engineering.

9. Budget and time

phase wall time Modal
0 MMLU-Pro, buried-state variant, SemIf cross-benchmarks 4 h ~$3
1 transformers 5 + row forward + hybrid smoke ½–1 day ~$2
2 controlled experiment ~1 day ~$25
3 promotion (+ MLX serving if the torch fallback is slow) ½–1 day ~$5
total ~3–4 days ~$37 of the remaining ~$240

10. Execution log and results (2026-09-20, branch qwen35)

Everything below ran on 2026-09-20 between 15:30 and 18:30. Spend: ~$95 of Modal H100 time (4B trials ≈ $4.50 each, 9B ≈ $7, plus probes, benches, locked reads and the ablation), plus $0.03 of Jev calls.

Phase 1 — port (done)

  • transformers 5.17 / peft 0.21 on the lock. The 62-test suite passes. fp32 parity of the published Kev-4B under transformers 5: max |Δp| = 2e-5 on 30 development questions against the saved H100 probabilities (runs/v7-rc3/01-trial-1/development/rows.json).
  • Row-batched forward (kev/model.py: rows_of, forward_rows_batch): on the attention-only Kev-4B it is bit-identical to the packed block-causal form (max |Δp| = 0.000000, 0 flips, 24 records, fp32 MPS) and 9% faster. Hybrid detection from config.layer_types; LoRA on in_proj_qkv/z/a/b, out_proj for DeltaNet layers.
  • Hybrid isolation and serving prefix path on Qwen3.5-0.8B: together-vs-alone and prefix-vs-full within 1e-5 (tests/test_model.py::test_hybrid_rows_isolation_and_prefix); cache replication via reorder_cache on DynamicCache(config=…) with LinearAttentionLayer states.
  • Modal image: flash-linear-attention + triton>=3.7.1 (fla refuses its gated chunk backward on Hopper with Triton 3.4–3.7.0, fla#640; the first smoke trial hit exactly that guard). Row-form training cost: 0.11 s/record at 4B, 0.18 s/record at 9B on one H100 (Kev-4B packed was ~0.06); 4B trial 61–63 min, 9B 88–100 min.
  • Mac latency is the cost of the hybrid. Same 5-question request, bf16, M5: Kev-4B 174 ms; Kev(3.5)-4B 779 ms (reference DeltaNet and causal-conv kernels; no MPS fla). 0.8B: 329 ms vs Qwen3-0.6B 123 ms. The prefix cache does not help on MPS at these sizes. MLX-LM (SemIf's path) is the fix if the Mac story matters; on CUDA the kernels are fast.

Phase 2 — controlled experiment (done): same data (decision-v7), same recipe, new base

transfer-v4 development (764 records), paired record-clustered bootstraps against the released checkpoint on the same items (scripts/compare_q35.py):

trial seed dev acc transfer acc paired Δ vs released [95% CI] deadline held-out pairs MMLU PAWS Emotion Brier conf. err cov@5% err
Kev-4B (Qwen3, released) 1 0.854 0.790 reference 0.53 0.73 0.65 0.72 0.66 0.328 0.082 0.31
Kev(3.5)-4B 0 0.877 0.788 −0.007 [−0.042, +0.028] 0.55 0.72 0.70 0.76 0.59 0.339 0.096 0.48
Kev(3.5)-4B 1 0.876 0.800 +0.015 [−0.024, +0.056] 0.53 0.75 0.71 0.76 0.59 0.322 0.095 0.52
Kev(3.5)-4B 2 0.877 0.794 +0.013 [−0.013, +0.046] 0.55 0.78 0.70 0.74 0.54 0.316 0.082 0.54
Kev(3.5)-4B 3 0.873 0.770 −0.024 [−0.064, +0.021] 0.50 0.69 0.64 0.76 0.59 0.346 0.093 0.49
Kev-8B (Qwen3, released) 0 0.863 0.796 reference 0.60 0.69 0.70 0.78 0.56 0.337 0.099 0.45
Kev(3.5)-9B 0 0.871 0.802 +0.004 [−0.046, +0.055] 0.70 0.75 0.74 0.78 0.57 0.308 0.069 0.54
Kev(3.5)-9B 1 0.876 0.812 +0.027 [−0.018, +0.064] 0.72 0.80 0.74 0.76 0.59 0.291 0.075 0.53
Jev – 0.845 0.857 0.93 0.86 0.90 0.79 0.59 0.211 0.037 0.70

Seed means: 4B 0.788 (4 seeds) vs Kev-4B's 0.778 (3 seeds); 9B 0.807 (2) vs Kev-8B's 0.785 (2).

Against the section-7 criteria, on the development partition: not met. No single seed beats its released counterpart with a CI excluding zero; deadline reached 0.70–0.72 at 9B, not 0.75. What did move, on every seed: in-distribution accuracy (+2.2 pp at 4B, +1.0 at 9B), held-out-pair correctness (4B 3 of 4 seeds ≥ 0.70, 9B both — the first time the original release screen is met at 8–9B), coverage at ≤ 5 % error (0.31 → ~0.50 at 4B; 0.45 → 0.53 at 9B), Brier and confident errors at 9B, MMLU (+4–6 pp at both sizes), and PAWS at 4B. Emotion is worse at 4B (−7 pp).

Locked test, read once per candidate (candidates selected on development only: 4B seed 2 = q35-4b-s23/00-trial-0, dev 0.877; 9B seed 1 = q35-9b/01-trial-1, dev 0.876). runs/locked/kev-4b-q35/, runs/locked/kev-9b-q35/:

in-distribution out-of-domain Brier held-out pairs conf. err paired Δ acc vs predecessor on the test items
Kev-4B (released) 0.856 0.806 0.294 0.66 0.066
Kev(3.5)-4B 0.870 0.832 0.266 0.72 0.069 +0.029 [−0.009, +0.064]
Kev-8B (released) 0.870 0.780 0.327 0.62 0.099
Kev(3.5)-9B 0.873 0.837 0.243 0.75 0.055 +0.073 [+0.028, +0.117]

On the confirmatory read the 9B beats Kev-8B by 7.3 pp with a CI excluding zero and a Brier 0.084 lower; the 4B beats Kev-4B by 2.9 pp with a CI that grazes zero. Both new checkpoints are above their predecessors on every column. This is one read, on the partition the criteria did not name; it is reported as such.

The deadline hypothesis: refuted as stated, and the reason is now known

The base's date arithmetic (Qwen3.5-9B 0.82, 4B 0.68 zero-shot) does not survive training: 9B trained 0.70–0.72, 4B 0.50–0.55. A diagnostic that reads the adapted backbone through the plain LM head with letter logits (scripts/base_mmlu_probe.py --adapter; runs/probes/qwen35-4b-base-base-adapted-q35-4b-s1-transfer-v4) gives deadline 0.53 — identical to the pointer-head number — while MMLU is retained (0.72). The skill is lost in the representation during LoRA training, not at the readout. Independently, issue #8 (3x3xX3N0N) shows that appending "The report arrived N days after/before the deadline" lifts Kev-4B from 0.575 to 0.963 with the unmodified head; reproduced here: Kev-4B 0.525 → 0.925, Kev(3.5)-4B → 1.000. A generic annotator (pairwise date differences without role names) helps far less (0.55 / 0.675), so there are two gaps: date subtraction (large) and binding a raw date fact to the policy's roles (smaller; the newer base does it better). Consequences: the ordinal readout drops to last; the next experiment is the training-data change (relational day counts and a date_facts field rendered in the date-bearing families), with issue #8's number as the ceiling. Ablation (q35-4b-nopolicy, Qwen3.5-4B, seed 1, same recipe): training on the ten public sources only gives deadline 0.50; public + rule trees without the date-bearing policy families gives 0.55; the full recipe 0.53; the untrained base 0.68. So the date-bearing families are not the cause — any LoRA fine-tune on this format erodes the base's date arithmetic, the same way it erodes MMLU. The fix is therefore not in the policy templates. Options, in order: give the model the number (issue #8's route, as training renderings + an optional request preprocessor), or retention methods that the earlier lr/anchor experiments did not solve. Side result of the same ablation: the rule trees are what teach compositional rules (held-out (A or B) and C 0.69 → 0.91, or not 0.44 → 0.75 when they are added), and the policy pairs add the rest (pairs 0.52 → 0.69 → 0.75).

Phase 0 — new evaluation columns (done)

transfer-v9 (evals/v9/transfer-v9, kev/transfer_v9.py): transfer-v4 byte-identical + 200 MMLU-Pro (10-way) + 80 buried-state + 110 unknowable / 110 intact controls per partition; mirrored to the Hub (SUITES_REVISION a957287d). New metrics in kev.metrics: coverage_at_5pct_error, coverage_at_1pct_error, and an unknowable block (mean max-probability and share ≥ 0.9 against the paired intact controls). Development partition:

model knowable acc MMLU-Pro buried (avg of 4) unknowable: mean max-p (intact) share ≥ 0.9 (intact) cov@5% err
Jev (live, $0.02) 0.854 0.840 0.70 0.61 (0.92) 0.09 (0.75) 0.70
Qwen3.5-9B base, untrained 0.696 0.540 0.64 0.52 (0.78) 0.00 (0.44) 0.38
Kev-4B 0.729 0.440 0.69 0.78 (0.96) 0.44 (0.90) 0.25
Kev-8B 0.747 0.500 0.72 0.77 (0.97) 0.26 (0.91) 0.41
Kev(3.5)-4B s2 0.742 0.500 0.66 0.72 (0.97) 0.19 (0.88) 0.47
Kev(3.5)-9B s1 0.771 0.545 0.74 0.67 (0.97) 0.05 (0.93) 0.46

MMLU-Pro separates the generations where 4-way MMLU did not (untrained: Qwen3-8B 0.380 vs Qwen3.5-9B 0.540), and the trained models retain it. On the unknowable family the Qwen3-based Kevs are confidently wrong (26–44 % of evidence-free items answered at ≥ 0.9); Kev(3.5)-9B is at 5 %, Jev at 9 %, untrained bases near 0. Training with hard labels makes a model commit; the fix is unknowable training records with uniform targets (soft-label support in kev.train), listed below.

Cross-benchmarks with the field (done)

  • SemIf-style untrained readout (Qwen3.5-4B instruct, their exact prompt) on our transfer-v4: 0.747, Brier 0.362 — stronger than our base probe (0.692) and the untrained row people will compare to. Kev-4B is +7.7 pp [+3.2, +12.2] over it; Kev(3.5)-4B seed 1 +5.3 pp. runs/probes/qwen35-4b-semif-transfer-v4.
  • SemIf's authored 144 + 108 perturbations (evals/external/semif-v1, scripts/freeze_semif.py), with live Jev: Jev 0.965 (perturbations 1.00/1.00/1.00); Kev-4B 0.847; Kev-8B 0.903; Kev(3.5)-4B 0.896; Kev(3.5)-9B 0.917 (perturbations 0.97/0.97/0.97). SemIf's untrained Qwen3.5-4B: 0.813. Their "3.8 pp behind Jev" was agreement on a different subset; on labelled items live Jev is 15 pp above their readout.
  • scienthoon's 900 tickets (evals/external/scienthoon-v1, their live Jev rows converted, scripts/freeze_scienthoon.py): queue / angry / priority(unknowable) — Jev 0.897 / 0.914 / 0.447 (ECE 0.105); Kev-4B 0.687 / 0.375 / 0.498; Kev-8B 0.924 / 0.515 / 0.381; Kev(3.5)-4B 0.928 / 0.794 / 0.402 (ECE 0.086); Kev(3.5)-9B 0.952 / 0.911 / 0.430 (ECE 0.082). The Qwen3 Kevs fail "The customer sounds angry." because the instruction is an assertion, not a question, and every ticket is complaint-shaped: Kev-4B calls 95 % of tickets angry (base rate 37 %). The Qwen3.5 checkpoints mostly fix it; assertion-style Noul instructions belong in the training renderings regardless.

Decision (open, for review)

The pre-registered development-partition criteria are not met; the single confirmatory locked-test read shows Kev(3.5)-9B +7.3 pp [+2.8, +11.7] over Kev-8B with much better calibration, and Kev(3.5)-4B +2.9 pp [−0.9, +6.4] over Kev-4B, both above their predecessors on every metric measured, on every external suite, and with the original held-out-pair screen met on both 9B seeds. Costs: 4.5× slower on a Mac until an MLX path exists; transformers 5 required. Recommendation: publish Kev(3.5)-4B and Kev(3.5)-9B as the new family (naming below), keep the Qwen3 checkpoints on the Hub as the previous generation, and state the criteria outcome plainly in the cards. Publishing is a release action and waits for approval.

Kev-0.8B (added 20:00): Qwen3.5-0.8B-Base with the 0.6B recipe (lr 1e-4), three seeds: transfer 0.622 / 0.634 / 0.643 (Kev-0.6B 0.620), dev 0.817–0.829 (0.801), held-out pairs 0.27–0.44 (0.08). Seed 2 released as jaredpalmer/kev-0.8b; locked test 0.827 / 0.668 vs Kev-0.6B 0.808 / 0.642, paired +4.8 pp [+0.2, +9.3]. The family is now Qwen3.5 throughout (0.8B / 4B / 9B); the Qwen3 checkpoints are kept for Mac latency and no longer developed.

Next experiments, in order: (1) date-family renderings with relational day counts + date_facts (issue #8; ceiling ~0.93–1.0 on deadline); (2) unknowable training records with uniform targets; (3) assertion-style Noul instructions; (4) MLX serving for the hybrid on Mac; (5) multilingual slices.

Status and deferred work

  • Modal CUDA/batched path and backbone-v1 study completed; MBP path retained.
  • v2 source policy, PAWS/SciQ conversion, contrastive prototype, and data-ablation study completed. Findings remain exploratory.
  • v3 minimal-pair/calibration/metric corrections and regression tests (tests, tests).
  • v3 compositional generator with truth-table and boundary tests (generator).
  • v3 frozen matched suites (decision-v3, transfer-v3); both arms share 3,000 public records and 448 synthetic records.
  • Modal smoke and the matched 0.6B/4B comparison, seed 0 (ledger). Paired, record-clustered bootstrap, transfer-v3 development: capacity (4B vs 0.6B, same data): +17.5 pp acc CI [+13.0, +22.1] on legacy data, +19.0 pp CI [+12.3, +25.0] on compositional data; data (compositional vs legacy, same backbone): +3.6 pp CI [-0.4, +7.8] at 0.6B, +5.1 pp CI [0.0, +9.9] at 4B; Brier -0.053 CI [-0.102, -0.005] at 0.6B. Held-out compositional structures, both siblings correct: 0.6B 3%/6%; 4B 45%/52%. Held-out authorization: 4B 100% both arms (0.6B 50%). Held-out deadline (3-level score) stays near chance for all cells. No cell passes the 70% held-out-pair screen; none is a locked-test candidate. Cost: 4 H100 trials, 0.6B ~4.5 min and 4B ~13.5 min wall each, admission bound $8.86.
  • Second seed for the four v3 cells; overnight: v4/v5/v6 suites, ~60 further trials, three research previews (0.6B / 4B / 8B) with one ungated locked read each. See "Overnight autoresearch" and the log below.
  • Deferred: option-order architecture experiments. Do not infer Jev's architecture from zero argmax flips.
  • Deferred: LLM-authored product scenarios, with a separate verification model and retained provenance; needs explicit API/budget decisions.
  • Deferred: 8B runs after the data-versus-capacity result, not as an automatic escalation.
  • Deferred: final release/model-card/Hub updates until generalization and calibration justify them.

Relevant code: suite builder, rule generator, experiment runner, benchmark, Modal app, v3 tests.

Autoresearch log

Maintained by kev.autoresearch; full table in runs/leaderboard.md. Selection uses development partitions only.

  • Qwen3-0.6B-Base incumbent (v4 suites): transfer 0.620, dev 0.801, seeds [2], knobs {"epochs": 2, "lr": 0.0001, "p_none_pair": 0.25}
  • Qwen3-4B-Base incumbent (v4 suites): transfer 0.775, dev 0.858, seeds [0], knobs {"epochs": 2, "lr": 5e-05, "p_none_pair": 0.25}
  • Qwen3-8B-Base incumbent (v4 suites): transfer 0.796, dev 0.863, seeds [0], knobs {"epochs": 2, "lr": 5e-05, "p_none_pair": 0.25}
round base trials best transfer best knobs incumbent after spend
auto-06b-r1 Qwen3-0.6B-Base 8/8 0.596 {"epochs": 2, "accum": 1, "perm_kl": 0.5, "p_none_pair": 0.25} 0.592 $9.82
auto-4b-r1 Qwen3-4B-Base 6/6 0.767 {"epochs": 2, "lr": 3e-05, "accum": 2, "perm_kl": 0.2, "p_none_pair": 0.25, "lora_targets": "all"} 0.767 $147.26