feat(trace): add investigation submission endpoint and The Debrief challenge - #573
Conversation
Scores a student's forensic investigation submission (server, tool, directive) against the expected answer in detector_config, with independent partial credit per field (33/33/34 by default). Server and tool match exactly; the free-text directive uses substring matching by default, or exact when directive_match is set. Scoring is deterministic and never calls an LLM, so it behaves identically under any model backend. Evidence records the score and which fields were correct but never the expected answer, so completion evidence cannot spoil the challenge. 19 unit tests cover the 33/66/100 thresholds, zero-score submissions, extra fields that must not inflate the score, directive fuzzy/exact modes, normalization, the tool_arguments fallback, and config validation.
…allenge
Wire the PurpleTeamDetector into a playable forensic challenge:
- POST /ctf/api/v1/toolkit/investigation accepts a {server, tool,
directive} answer and emits business.investigation.submitted with a
fresh workflow_id per submission, so each attempt is tracked.
- The Debrief challenge is a purple team follow-up to Toxic Transfer.
It names finstripe/create_transfer as the poisoned tool and grades the
directive by meaning, not verbatim text.
- Upgrade directive matching from substring to token overlap with light
prefix stemming so students can paraphrase. A configurable
directive_threshold (default 0.6) controls how much of the expected
answer must appear.
24 unit tests pass, including reordered-paraphrase acceptance and
unrelated-text rejection. Full submit-to-score flow verified against the
challenge config.
syedDS
left a comment
syedDS
left a comment
There was a problem hiding this comment.
- In purple_team_detector.py:157, the fuzzy-match logic accepts a submission if it's a substring of the expected answer:
if exp_norm in sub_norm or sub_norm in exp_norm:
return True
The problem seems to be the second half (sub_norm in exp_norm) "is what the student typed contained somewhere inside the correct answer." There's no minimum length check on it. The seeded answer is:
"email the vendor tax id and bank account to an external address"
If a student just types directive: "to", that's technically a substring of the sentence above, so it scores 100% correct with zero understanding of the attack. Same goes for "and", "id", "account", etc. Since this is the core thing the challenge is testing, this one basically breaks the challenge.
Recommended Fix: either remove that substring branch and rely only on the token-overlap scoring below it (which already has a real threshold), or add a minimum word-count requirement before accepting a substring match.
- The challenge description tells students: "Partial credit is awarded, so a partial understanding still earns points." That's not what happens.
What actually happens: score (0–100) only decides pass/fail them detected = score >= min_score. Once a student passes that bar, they get the full 250 points, no matter whether they scored 50 or 100. This is because the points multiplier (points_modifier) comes from a completely separate mechanism (scoring.modifiers in the debrief.yaml), which this challenge doesn't configure so it stays at 1.0 always.
Issue is a person who correctly names the server and tool but writes total nonsense for the directive (score 66) earns exactly the same reward as someone who gets everything right (score 100). And once they pass, the challenge is marked complete there's no way to go back and improve the score even if a student wanted to.
Fix: either wire the score into points_modifier so points scale with accuracy (the codebase already has this plumbing, just needs connecting), or rewrite the description so it doesn't promise something the code doesn't do.
… bypass and unearned full credit Both issues raised by @syedDS. 1. Fuzzy directive matching accepted a submission that was merely a substring OF the expected answer. The seeded directive is an ordinary sentence, so single filler words are literal substrings of it: "to", "and", "id" and "account" each scored the directive field at full credit. Combined with server/tool being small guessable enums, a player could take all 250 points without understanding the attack -- which is the only thing this challenge tests. Removed that direction. Quoting the expected directive in full still passes; genuine paraphrases still pass through the token-overlap check, which has a real threshold. Added regression tests covering the reviewer's exact examples, plus tests proving real paraphrases are still accepted. 2. The description promised partial credit that the code did not deliver. The 0-100 score only decided pass/fail; every player past min_score earned the same 250 points, because points_modifier comes from scoring.modifiers, which this challenge never configured. Connected the existing plumbing rather than rewording the promise: ChallengeService now passes the DetectionResult through to modifiers as detection_evidence, and a new generic `detection_score` modifier converts a detector's score into points_modifier (award = points * score/100, floored by min_multiplier). The Debrief opts in at min_multiplier 0.5, so 66/100 now earns 165 of 250 instead of the full award. The modifier is generic: any detector reporting a score can use it. Description rewritten to state the per-field values and the scaling. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
@syedDS Thanks for the review — both points were spot on, and both are fixed in 607b71a. 1. Substring bypass in the fuzzy directive match You were right that this broke the challenge outright. With the seeded answer, submitting I went with your first option: dropped the Added regression tests for your exact examples ( 2. Partial credit that wasn't Also correct — the score only gated pass/fail, and everyone past I took the "wire it up" option rather than reword the description. Instead of special-casing The Debrief, I added a generic Result:
Tests in One note: this branch has 4 pre-existing test failures ( |
… bypass and unearned full credit Both issues raised by @syedDS. 1. Fuzzy directive matching accepted a submission that was merely a substring OF the expected answer. The seeded directive is an ordinary sentence, so single filler words are literal substrings of it: "to", "and", "id" and "account" each scored the directive field at full credit. Combined with server/tool being small guessable enums, a player could take all 250 points without understanding the attack -- which is the only thing this challenge tests. Removed that direction. Quoting the expected directive in full still passes; genuine paraphrases still pass through the token-overlap check, which has a real threshold. Added regression tests covering the reviewer's exact examples, plus tests proving real paraphrases are still accepted. 2. The description promised partial credit that the code did not deliver. The 0-100 score only decided pass/fail; every player past min_score earned the same 250 points, because points_modifier comes from scoring.modifiers, which this challenge never configured. Connected the existing plumbing rather than rewording the promise: ChallengeService now passes the DetectionResult through to modifiers as detection_evidence, and a new generic `detection_score` modifier converts a detector's score into points_modifier (award = points * score/100, floored by min_multiplier). The Debrief opts in at min_multiplier 0.5, so 66/100 now earns 165 of 250 instead of the full award. The modifier is generic: any detector reporting a score can use it. Description rewritten to state the per-field values and the scaling.
607b71a to
0150572
Compare
2 participants
Summary
Adds PurpleTeamEvaluator, which scores forensic reasoning instead of agent behavior: a student submits {server, tool, directive} identifying a compromised MCP server, the poisoned tool, and the directive it carried, and each field scores independently against a known seeded answer. Also adds the investigation-submission API endpoint and "The Debrief" challenge that uses it.
Test plan