Skip to content

Add OWASP Agent Memory Guard and AgentThreatBench to ASI06 tools - #321

Open
vgudur-dev wants to merge 4 commits into
GenAI-Security-Project:mainfrom
vgudur-dev:amg-asi06-tools
Open

vgudur-dev wants to merge 4 commits into
GenAI-Security-Project:mainfrom
vgudur-dev:amg-asi06-tools

Conversation

@vgudur-dev

Copy link
Copy Markdown

What this PR changes

Adds two open-source entries to ASI06 in data/tools-supplement.json, and the regenerated data/entries/ASI06.json and docs/data.js from npm run build:

No other entries change.

Type of change

  • Update to existing mapping (content, controls, CVE refs)

Source / evidence

Checklist

  • All referenced vulnerability IDs are valid (ASI06)
  • All external URLs checked manually (GitHub URLs return 200, see note on the ATLAS link)
  • Ran node scripts/validate.js locally and it passes
  • npm run build and npm test pass locally (137 tests)

Notes for reviewers

Disclosure: I'm a co-lead of Agent Memory Guard and I contributed AgentThreatBench to Inspect Evals, so please weigh these entries with that in mind. Neither tool covers all of ASI06. AMG's detectors are pattern based and miss some memory-poisoning techniques, which is part of why the benchmark is listed separately. I'm happy to drop either entry or change the wording if it doesn't fit the supplement's scope.

Signed-off-by: vgudur-dev <gudur.vaishnavi@gmail.com>
Signed-off-by: vgudur-dev <gudur.vaishnavi@gmail.com>
Signed-off-by: vgudur-dev <gudur.vaishnavi@gmail.com>
@emmanuelgjr

Copy link
Copy Markdown
Contributor

Thanks, @vgudur-dev. The disclosure and the honest note about coverage limits are appreciated.

What checks out

  • The ATLAS citation is accurate. ATLAS-2026.09.yaml mitigation AML.M0031 "Memory Hardening" names OWASP Agent Memory Guard as an open-source tool that implements memory-hardening controls.
  • Both entries meet the inclusion bar: open source (Apache-2.0 and MIT), actively maintained, and relevant to ASI06.
  • data/tools-supplement.json is the right source file, and the data/entries/ASI06.json and docs/data.js additions match it.

Before it can merge

  1. CI hasn't run yet. This is a fork PR, so a maintainer has to approve the workflow run. That's on our side.
  2. The ASI06 tagging needs a maintainer sign-off. Per project rules, risk mappings are confirmed by a human SME. That's routine, and not a request to change anything.
  3. Please squash on merge. The three commits share one message.

One suggestion, not a blocker
AgentThreatBench is an evaluation suite, and docs/TRIAGE_RULES.md rule 3 catalogues benchmarks in evals/EXTERNAL_BENCHMARKS.md. A catalogue row there (release, licence, coverage of ASI01 and ASI06) would make it findable alongside the other benchmarks. It can stay in the tools supplement too, and we can add the catalogue row in a follow-up if you'd rather keep this PR small.

Added AgentThreatBench to evaluate LLM agents' performance against adversarial content while maintaining utility and security.

Signed-off-by: vgudur-dev <gudur.vaishnavi@gmail.com>
@vgudur-dev

Copy link
Copy Markdown
Author

Thank you for the review and for confirming the citation. I took your suggestion and added AgentThreatBench to evals/EXTERNAL_BENCHMARKS.md in this PR, with a changelog line. One note for you: that file says its figures are transcribed from each paper's abstract, and AgentThreatBench has no paper, so the scale column comes from its README. The row also states that I contributed the benchmark and co-lead Agent Memory Guard. Happy to reword either if you'd prefer.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants