Skip to content

Surface LLM partial failures as user-visible warnings in scan results#207

Draft
Copilot wants to merge 3 commits into
masterfrom
copilot/fix-llm-failures-feedback
Draft

Surface LLM partial failures as user-visible warnings in scan results#207
Copilot wants to merge 3 commits into
masterfrom
copilot/fix-llm-failures-feedback

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Apr 6, 2026

LLM analysis nodes (summary_generation_node, impact_analysis_node, privacy_compliance_node) silently swallow exceptions and set their outputs to None, leaving scan results with empty sections and no explanation to the user.

Backend

  • WorkflowState — adds llm_warnings: Optional[list] field to carry accumulated failure messages through the pipeline
  • nodes.py — each LLM node reads the current warnings list, appends a message if its output is None, and writes the updated list back to state:
    warnings = list(state.get("llm_warnings") or [])
    if executive_summary is None:
        warnings.append("Summary unavailable — LLM service temporarily failed")
    return Command(goto=..., update={"executive_summary": executive_summary, "llm_warnings": warnings})
  • api/main.py — exposes partial_failures in the scan result response (empty list on full success, warning strings on partial failure)

Frontend

  • SummaryPanel.jsx — reads rawScanResult.partial_failures and renders an amber warning banner above summary content in every display branch; accessible via role="alert" with a descriptive aria-label
  • SummaryPanel.scss — styles for the banner with dark- and light-theme variants

Tests

  • 9 new unit tests in tests/workflow/test_nodes_llm_warnings.py covering: no warning on success, generic exception, LLMFallbackError, and multi-node accumulation across all three LLM nodes

Copilot AI and others added 2 commits April 6, 2026 22:38
- Add `llm_warnings` to WorkflowState to accumulate failure messages
- Update summary_generation_node, impact_analysis_node, and
  privacy_compliance_node to append a human-readable warning to
  `llm_warnings` whenever their LLM call fails
- Include `partial_failures` list in scan result API response so
  consumers can display which sections are unavailable
- Show a yellow warning banner in SummaryPanel when `partial_failures`
  is non-empty (e.g. "Summary unavailable — LLM service temporarily failed")
- Add CSS for the warning banner (dark + light theme)
- Add 9 unit tests covering warning accumulation across all three nodes"

Agent-Logs-Url: https://github.com/Stanzin7/ExtensionShield/sessions/8a1fb240-97cb-498e-aced-9c548c22f375

Co-authored-by: Stanzin7 <109467334+Stanzin7@users.noreply.github.com>
- Initialize llm_warnings fixture as [] instead of None in tests
- Explicitly verify pre-existing warning is preserved unchanged in accumulation test
- Improve aria-label on warning banner for better screen reader support

Agent-Logs-Url: https://github.com/Stanzin7/ExtensionShield/sessions/8a1fb240-97cb-498e-aced-9c548c22f375

Co-authored-by: Stanzin7 <109467334+Stanzin7@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix LLM failures that return empty results without user feedback Surface LLM partial failures as user-visible warnings in scan results Apr 6, 2026
Copilot AI requested a review from Stanzin7 April 6, 2026 22:40
@github-actions github-actions Bot added the area: backend Changes to the Python backend and scanning pipeline label Apr 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: backend Changes to the Python backend and scanning pipeline

Projects

None yet

Development

Successfully merging this pull request may close these issues.

LLM failures silently return empty results without user feedback

2 participants