Surface LLM partial failures as user-visible warnings in scan results#207
Draft
Copilot wants to merge 3 commits into
Draft
Surface LLM partial failures as user-visible warnings in scan results#207Copilot wants to merge 3 commits into
Copilot wants to merge 3 commits into
Conversation
- Add `llm_warnings` to WorkflowState to accumulate failure messages - Update summary_generation_node, impact_analysis_node, and privacy_compliance_node to append a human-readable warning to `llm_warnings` whenever their LLM call fails - Include `partial_failures` list in scan result API response so consumers can display which sections are unavailable - Show a yellow warning banner in SummaryPanel when `partial_failures` is non-empty (e.g. "Summary unavailable — LLM service temporarily failed") - Add CSS for the warning banner (dark + light theme) - Add 9 unit tests covering warning accumulation across all three nodes" Agent-Logs-Url: https://github.com/Stanzin7/ExtensionShield/sessions/8a1fb240-97cb-498e-aced-9c548c22f375 Co-authored-by: Stanzin7 <109467334+Stanzin7@users.noreply.github.com>
- Initialize llm_warnings fixture as [] instead of None in tests - Explicitly verify pre-existing warning is preserved unchanged in accumulation test - Improve aria-label on warning banner for better screen reader support Agent-Logs-Url: https://github.com/Stanzin7/ExtensionShield/sessions/8a1fb240-97cb-498e-aced-9c548c22f375 Co-authored-by: Stanzin7 <109467334+Stanzin7@users.noreply.github.com>
Copilot
AI
changed the title
[WIP] Fix LLM failures that return empty results without user feedback
Surface LLM partial failures as user-visible warnings in scan results
Apr 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
LLM analysis nodes (
summary_generation_node,impact_analysis_node,privacy_compliance_node) silently swallow exceptions and set their outputs toNone, leaving scan results with empty sections and no explanation to the user.Backend
WorkflowState— addsllm_warnings: Optional[list]field to carry accumulated failure messages through the pipelinenodes.py— each LLM node reads the current warnings list, appends a message if its output isNone, and writes the updated list back to state:api/main.py— exposespartial_failuresin the scan result response (empty list on full success, warning strings on partial failure)Frontend
SummaryPanel.jsx— readsrawScanResult.partial_failuresand renders an amber warning banner above summary content in every display branch; accessible viarole="alert"with a descriptivearia-labelSummaryPanel.scss— styles for the banner with dark- and light-theme variantsTests
tests/workflow/test_nodes_llm_warnings.pycovering: no warning on success, generic exception,LLMFallbackError, and multi-node accumulation across all three LLM nodes