Always surface both Stage-1 (interception) and Stage-2 (judged) scores; raw interception overcounts ~2×
The two-stage design (HTTP interception → LLM judge) is a genuine strength, but leaderboard/README numbers should always report both stages separately, because raw Stage-1 interception overcounts success ~2×:
| Model (v2) |
raw intercepted |
judge-confirmed |
precision |
| gemini-3.1-pro-preview |
69 |
37 |
54% |
| gemini-3.5-flash |
66 |
33 |
50% |
| aggregate |
156 |
78 |
50% |
Reporting only interception makes agents look ~2× better than they are (the "right request, wrong intent" gap — e.g. an agent that hits the target endpoint with wrong parameters). Conversely, reporting only the judged number hides the interception coverage.
Ask: every leaderboard row and headline shows intercepted% AND judged% (with the judge model named). This is the empirical justification for the two-stage design and prevents the metric from being misread. Related to but distinct from the phantom-numbers issue (#221).
Always surface both Stage-1 (interception) and Stage-2 (judged) scores; raw interception overcounts ~2×
The two-stage design (HTTP interception → LLM judge) is a genuine strength, but leaderboard/README numbers should always report both stages separately, because raw Stage-1 interception overcounts success ~2×:
Reporting only interception makes agents look ~2× better than they are (the "right request, wrong intent" gap — e.g. an agent that hits the target endpoint with wrong parameters). Conversely, reporting only the judged number hides the interception coverage.
Ask: every leaderboard row and headline shows
intercepted%ANDjudged%(with the judge model named). This is the empirical justification for the two-stage design and prevents the metric from being misread. Related to but distinct from the phantom-numbers issue (#221).