Repository navigation
Add OWASP Agent Memory Guard and AgentThreatBench to ASI06 tools - #321
vgudur-dev wants to merge 4 commits into
Conversation
Signed-off-by: vgudur-dev <gudur.vaishnavi@gmail.com>
Signed-off-by: vgudur-dev <gudur.vaishnavi@gmail.com>
Signed-off-by: vgudur-dev <gudur.vaishnavi@gmail.com>
|
Thanks, @vgudur-dev. The disclosure and the honest note about coverage limits are appreciated. What checks out
Before it can merge
One suggestion, not a blocker |
Added AgentThreatBench to evaluate LLM agents' performance against adversarial content while maintaining utility and security. Signed-off-by: vgudur-dev <gudur.vaishnavi@gmail.com>
|
Thank you for the review and for confirming the citation. I took your suggestion and added AgentThreatBench to |
What this PR changes
Adds two open-source entries to
ASI06indata/tools-supplement.json, and the regenerateddata/entries/ASI06.jsonanddocs/data.jsfromnpm run build:memory_poisontask mapped to ASI06)No other entries change.
Type of change
Source / evidence
agent_threat_bench_memory_poisonto ASI06 (Memory and Context Poisoning). The suite currently covers 2 of the 10 OWASP Agentic Top 10 categories (ASI01 and ASI06).Checklist
node scripts/validate.jslocally and it passesnpm run buildandnpm testpass locally (137 tests)Notes for reviewers
Disclosure: I'm a co-lead of Agent Memory Guard and I contributed AgentThreatBench to Inspect Evals, so please weigh these entries with that in mind. Neither tool covers all of ASI06. AMG's detectors are pattern based and miss some memory-poisoning techniques, which is part of why the benchmark is listed separately. I'm happy to drop either entry or change the wording if it doesn't fit the supplement's scope.