Recruit multi-user testing across numerical and conceptual problem types

[aaronashby]

Overview

Judge/Sonnet-tutor correctness bugs keep surfacing from manual single-tester testing even after prior fixes seemed complete — e.g. #50, and the earlier solve-phase "coincidental value match" bug. Testing so far has been narrow: a small number of numeric problems (fraction addition, function evaluation) worked through by one person. Recruit other testers to run the tutor across a wider variety of problems and record what they find, rather than relying on one person occasionally poking at it.

Goals

• Problem-type coverage: numerical problems (a single correct value — arithmetic, algebra) and conceptual/open-ended problems (proofs, word problems, "explain why" style questions with no single final-answer string) — the judge and system prompt have mostly been exercised against the numerical case so far.
• Phase coverage: gap_check scaffolding (including intentionally answering intermediate sub-questions, per gap_check resolves early when a scaffolded sub-question is answered correctly #50), solve scaffolding, the final-answer toggle, and misconception classification (deliberately careless vs. genuine wrong answers).
• Consistent reporting: each finding filed as its own issue (repro steps + observed vs. expected — gap_check resolves early when a scaffolded sub-question is answered correctly #50 is a reasonable template), not bundled together, so they can be triaged and prioritized independently.

Suggested approach

• Seed a handful of test problems spanning both types across one or two courses (a couple of numeric ones already exist from recent manual testing).
• Give testers a short script to work through per problem: get a gap-check question wrong, then right; try answering an intermediate scaffold sub-question as if it were the final check-in; get a solve sub-step "right" without flagging it as the final answer; try free-text/conceptual answers where there's no single correct value.
• Collect findings as individual GitHub issues rather than one running thread.