Skip to content

matchWeakness fails open to "novel" on error, writing a duplicate row instead of just losing a data point #54

Description

@aaronashby

Overview

matchWeakness (app/queries/claude.ts) returns "novel" on any failure — API error, malformed response, or an out-of-range match index. Its docstring frames this as mirroring classifyMisconception's "never let a flaky model corrupt state" bias, but the two failure modes are not the same risk class:

  • classifyMisconception failing returns null, and runMisconceptionPipeline simply skips the write. Worst case: one missed data point.
  • matchWeakness failing returns "novel", which runMisconceptionPipeline treats as "no existing match, insert a new row." Worst case: a duplicate weakness row written and persisted with no reconciliation job to ever merge it back.

A Haiku timeout or rate-limit blip during a busy grading period would currently cause every in-flight wrong answer's dedup check to silently start duplicating weakness rows, rather than just skipping the (comparatively harmless) increment.

Suggested fix

Give matchWeakness a third outcome distinct from both { id } and "novel" — e.g. "unknown" — for its failure path, and have runMisconceptionPipeline treat "unknown" the same way it already treats a null from classifyMisconception: skip the write, log it, move on. This makes the actual behavior match what the docstring already claims instead of asserting parity while writing wrong data.

Notes

Metadata

Metadata

Assignees

No one assigned

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions