Repository navigation
Watcher backlog triage (2026-10): 47 closed, rulings needed on benchmarks and 31 CVE candidates #324
Description
Summary
On 2026-10-28, the 98 open automated-watch issues (opened 2026-10-05 → 10-26) were routed against docs/TRIAGE_RULES.md. Full arXiv abstracts and NVD records were read for every item.
- 47 closed as rule 1, note and close, with each reason posted on the issue. One is a CVE ([new-cve] CVE-2026-108957 — A security flaw has been discovered in TinyLLaVA TinyLLaVA_Factory up to 1abc6ac #296 TinyLLaVA: no release version is named, only a commit hash).
- 4 are already routed by open PRs: [new-research] arXiv:2610.03434 — Persona Guardrail: A Production-Grade Defense Framework for Agentic Systems #194, [new-research] arXiv:2610.03153 — EvoRiskBench: An Evolving Benchmark for Runtime Security Risks in Workspace Agen #196 and [new-research] arXiv:2610.03089 — Securing Computer-Use Agents Against Branch Steering Attacks #198 by Triage: catalogue EvoRiskBench, PAGE and STEER-Bench (#196, #194, #198) #222, and [new-cve] CVE-2026-51871 — Devika v1.0 is vulnerable to Code Injection in the Runner.execute function in sr #212 by incidents: add INC-148, Devika Runner.execute runs LLM-generated commands (CVE-2026-51871) #223. They close when those PRs merge.
- 47 remain, listed below. Watcher: one digest issue per run for arXiv and NVD #322 stops the volume recurring: one digest issue per source per week.
No risk mapping, severity or control failure is proposed here (C4).
Decisions needed
- Benchmarks (7). Rule 3 is ratified, so these can go into
evals/EXTERNAL_BENCHMARKS.mdwith the OWASP column marked DRAFT. None of the abstracts states a release or licence, so each row would record "not released" or "none stated" unless the release is found. Recommend: catalogue all 7 in one PR. - Blended (2): [new-research] arXiv:2610.13227 — Understanding and Mitigating Cross-Session Attacks via Persistent Artifacts in A #228 SkillRelayBench and [new-research] arXiv:2610.23109 — Trojan Horse: Automated Repository Poisoning against Coding Agents #283 TrojanBench. Triage: catalogue EvoRiskBench, PAGE and STEER-Bench (#196, #194, #198) #222 records the precedent for [new-research] arXiv:2610.03434 — Persona Guardrail: A Production-Grade Defense Framework for Agentic Systems #194 and [new-research] arXiv:2610.03089 — Securing Computer-Use Agents Against Branch Steering Attacks #198: when a defense paper's secondary contribution is a named benchmark, catalogue it and name the authors' defense. Recommend: apply the same precedent once Triage: catalogue EvoRiskBench, PAGE and STEER-Bench (#196, #194, #198) #222 merges.
- CVE incident candidates (31 across 5 products).
- Is Docling GenAI tooling? It calls itself a document parser for the generative AI ecosystem, but its 10 flaws are generic parsing bugs.
- One record per CVE, or one per product advisory batch? vLLM has 9 CVEs and Pydantic AI has 10.
- Recommend: one record per product per disclosure batch, citing every CVE id. Docling is included only if you rule it is GenAI tooling.
- Needs a ruling (7). These are in the table below. Each states the conflict.
Rule 3 — benchmark catalogue candidates (7)
| Issue | Target | Evidence |
|---|---|---|
| #226 | APEX-Harm | Introduces APEX-Harm, "a benchmark for evaluating prompt and script injection against long-horizon agents", 180 task instances per model across six attack categories. |
| #279 | MobileInject | Introduces MobileInject, "the first benchmark for evaluating mobile agents against multimodal injection attacks", with 108 executable risk instances across 11 scenarios. |
| #280 | MobileJailBench | Presents MobileJailBench, "the first evolving jailbreak benchmark for mobile agents", 520 tasks across 14 risk categories. |
| #300 | NTBench | Introduces NTBench, the Near-Target Benchmark for prompt injection that seeks plausible-but-incorrect or harmful outcomes, built on legitimate tasks from existing datasets. |
| #310 | AgentPhishing | Introduces AgentPhishing, "a benchmark for evaluating whether persuasive phishing attacks can evade detection and induce LLM email agents to comply", evaluated on nine agents (also contributes the PhishOpt attack). |
| #311 | AfterDeleteBench | Introduces AfterDeleteBench, "a controlled behavioral audit" of post-deletion information persistence in LLM agent memory. |
| #315 | Omni-UBench | Presents Omni-UBench, a framework for omni-modal guardrail assessment curating "over 183k samples drawn from 35+ heterogeneous sources" with safety labels. |
Blended — defense/attack paper that also introduces a named benchmark (2)
| Issue | Target | Evidence |
|---|---|---|
| #228 | SkillRelayBench | Characterises cross-session skill attacks, constructs SkillRelayBench (270 adversarial cases) and proposes the SkillRelayGuard defense; defense paper with a named benchmark as a secondary contribution. |
| #283 | TrojanBench | Primary contribution is TrojanForge, an automated repository-poisoning attack framework; also constructs TrojanBench (384 poisoned repositories) as a secondary contribution. |
Needs a ruling (7)
| Issue | Target | Evidence |
|---|---|---|
| #225 | Fuzzing framework (Rule 1) whose evaluation found verified CWE-class bugs in 343 tools across 118 of 273 real open-source MCP servers; conflicts with worked examples INC-114 and INC-126, where MCP-server mass-scan findings were recorded as incidents. | |
| #232 | INC-137 | Self-described "conceptual research paper" proposing the PASAC framework (Rule 1), but it is a comparative case study of real 2026 incidents, including OpenAI agents compromising Hugging Face production (already INC-137) plus Anthropic and Gemini out-of-scope cases with no record; Rule 1 vs Rule 2 conflict. |
| #241 | Primarily an architecture (Rule 1) for investigating candidate autonomous-agent activity, but reports public traces resolved to "specific Census requests from 16 - 17 June"; unclear whether this documents a real-world GenAI event (no incident record found), and the authors state traces "do not establish organizational attribution". | |
| #254 | JeecgBoot is a general low-code platform; the flaw is missing authorization in its AIRAG promptExperiment handler that lets any user run LLM evaluation runs, so whether it counts as GenAI tooling is a scope call (versioned, VulnCheck advisory). | |
| #276 | ABF-Bench | Introduces ABF-Bench, a verifiable benchmark for agentic blockchain forensics (Rule 3 on its face), but it measures fund-tracing capability, not a GenAI security risk, so it may fall outside EXTERNAL_BENCHMARKS.md scope. |
| #302 | Detection technique (Rule 1) whose evaluation on 13,598 real ClawHub skills found 612 insecure compositions; conflicts with worked examples INC-118/INC-121, where scans of live skill registries were recorded as incidents. | |
| #319 | Goose terminal integration sends unredacted shell history to LLM providers; Red Hat advisory exists, but neither NVD nor Red Hat names an affected Goose version (Red Hat lists only the RHEL 9/10 goose package as Affected). |
CVE incident candidates (31)
Each names an affected version and has a citable advisory per NVD. No incident record cites any of these CVE ids yet.
| Product | Issues | CVEs | CNA |
|---|---|---|---|
| vLLM | #245, #246, #247, #248, #249, #250, #251, #252, #253 | 9 | security-advisories@github.com |
| Docling | #255, #256, #257, #258, #259, #260, #261, #262, #263, #264 | 10 | security-advisories@github.com |
| Pydantic AI | #265, #266, #267, #268, #269, #270, #271, #272, #273, #274 | 10 | security-advisories@github.com |
| 9router (Hermes Agent settings endpoint) | #275 | 1 | disclosure@vulncheck.com |
| MLflow | #320 | 1 | security-advisories@github.com |
🤖 Generated with Claude Code
Activity