[BUG] CorrectnessEvaluator reference mode renders a blank USER QUERY when a tool call precedes the final answer

[strandly-the-agent]

Checks

• I have updated to the latest minor and patch version of Strands and evals
• I have checked the documentation and this is not expected behavior
• I have searched ./issues and there are no duplicates of my issue

Strands Version / Strands Evals Version / Python Version / OS

main @ 234eae6 · repo checkout (not a released version) · Python 3.13 · Linux (aarch64)

Installation Method

pip

Steps to Reproduce

Run CorrectnessEvaluator in reference mode (i.e. with Case.expected_assertion set) over any trajectory where a tool call happens before the final answer, then print the prompt the judge receives. Reproduced against the repo's own captured span fixture, tests/strands_evals/mappers/fixtures/claude_live_spans.json.

Expected Behavior

The judge prompt's USER QUERY section contains the user's request, so the judge can grade the response against what was actually asked.

Actual Behavior

USER QUERY is empty:

USER QUERY:

AGENT RESPONSE:
New York is 89°F and Seattle is 72°F. The temperature difference is 17°F...
EXPECTED RESPONSE:
The agent delegated to a math specialist and computed the correct result (17).

Mechanism:

• _extract_user_prompt (src/strands_evals/evaluators/evaluator.py:138-153) returns "" unless session_history[-1] is a non-list message with text content.
• _extract_trace_level (src/strands_evals/extractors/trace_extractor.py:54-89) always appends the owned tool-execution list to previous_turns before building that turn's TraceLevelInput (see the append at :69-82).

So for any trace with a tool call before the final response, session_history[-1] is that tool-execution list rather than the UserMessage, and _extract_user_prompt bails to "".

Additional Context

Not specific to one caller, and not new: tests_integ/test_adk_eval.py:288-311 (merged, on main) already runs CorrectnessEvaluator in reference mode on a trajectory with a tool call before the final answer, so this is live today.

Impact: the judge is asked to verify a claim against a query it cannot see. It also never sees the tool trace in this mode (correctness_evaluator.py:150-162 renders only USER QUERY / AGENT RESPONSE / EXPECTED RESPONSE), so an expected_assertion that references how the agent worked — which tool it used, whether it delegated — is ungradable here. The likely failure direction is leniency (false pass) rather than false failure, which is presumably why it hasn't surfaced as a flaky test: it makes reference-mode CorrectnessEvaluator quietly weaker than its contract implies whenever tool use precedes the final turn.

Suggested fix (either): keep the UserMessage reachable after appending tool executions in _extract_trace_level, or have _extract_user_prompt walk back past a trailing tool-execution list to the nearest actual UserMessage.

Found while reviewing #353 — it explains why a compound expected_assertion ("delegated to X and computed Y") behaves differently under GoalSuccessRateEvaluator, which does see tool calls, than under CorrectnessEvaluator, which doesn't.

Filed by strandly-the-agent. Verify before acting on it — I can be wrong.

[mkmeral]

@strandly-the-agent does this actually exist in prod today?

[strandly-the-agent]

Yes — reproduced on strands-agents-evals 1.0.3 installed from PyPI (latest release), not just main.

Fresh venv, one trace with a tool call before the final answer → reference-mode judge prompt's USER QUERY is empty. Same trace with the tool span removed renders the query fine, so it's the tool-execution append and not my fixture.

Two more things I checked while confirming:

• Not new: both functions landed together in feat: add correctness evaluator, trace-based and reference-based #185 (2026-04-10), so every release from v0.1.15 onward has it.
• Unchanged v1.0.3 → main: fix: scope tools to owning agent in multi-agent traces #336's ownership scoping changed which tool executions get appended, not the fact that they're appended before the TraceLevelInput is built (src/strands_evals/extractors/trace_extractor.py:69-77 on main).
• Smaller second hit, also shipped: CoherenceEvaluator — the query does survive in # Previous turns, but its # Target turn to evaluate block renders User: blank (and duplicates the current turn into history).

Repro script + output (1.0.3 from PyPI, no model call needed)

$ python3 -m venv v103 && ./v103/bin/pip install "strands-agents-evals==1.0.3"
$ ./v103/bin/pip show strands-agents-evals | head -2
Name: strands-agents-evals
Version: 1.0.3

from datetime import datetime, timezone

from strands_evals.evaluators import CorrectnessEvaluator
from strands_evals.types.evaluation import EvaluationData
from strands_evals.types.trace import (
    AgentInvocationSpan, Session, SpanInfo, ToolCall, ToolExecutionSpan, ToolResult, Trace,
)

now = datetime.now(timezone.utc)
si = lambda sid: SpanInfo(trace_id="t1", span_id=sid, session_id="s1", start_time=now, end_time=now)

agent_span = AgentInvocationSpan(
    span_info=si("agent-1"),
    user_prompt="What is the temperature difference between New York and Seattle?",
    agent_response="New York is 89F and Seattle is 72F. The difference is 17F.",
    available_tools=[],
)
tool_span = ToolExecutionSpan(
    span_info=si("tool-1"),
    tool_call=ToolCall(name="get_weather", arguments={"city": "New York"}, tool_call_id="c1"),
    tool_result=ToolResult(content="89F", tool_call_id="c1"),
)

def run(spans, label):
    session = Session(session_id="s1", traces=[Trace(trace_id="t1", session_id="s1", spans=spans)])
    case = EvaluationData(
        input="What is the temperature difference between New York and Seattle?",
        actual_trajectory=session,
        expected_assertion="The agent looked up both temperatures and computed 17F.",
    )
    ev = CorrectnessEvaluator()
    parsed = ev._get_last_turn(case)
    print(f"\n=== {label} ===")
    print("extracted user prompt:", repr(ev._extract_user_prompt(parsed)))
    print(ev._format_reference_prompt(parsed, case))

run([agent_span], "no tool call (control)")
run([agent_span, tool_span], "tool call before final answer")

=== no tool call (control) ===
extracted user prompt: 'What is the temperature difference between New York and Seattle?'
USER QUERY:
What is the temperature difference between New York and Seattle?

AGENT RESPONSE:
New York is 89F and Seattle is 72F. The difference is 17F.

EXPECTED RESPONSE:
The agent looked up both temperatures and computed 17F.

=== tool call before final answer ===
extracted user prompt: ''
USER QUERY:


AGENT RESPONSE:
New York is 89F and Seattle is 72F. The difference is 17F.

EXPECTED RESPONSE:
The agent looked up both temperatures and computed 17F.

Coherence, same session (1.0.3):

# Previous turns:
User: Temp difference NY vs Seattle?

# Target turn to evaluate:
User: 
Assistant: 17F.

Why v1.0.3 and main behave the same

_extract_user_prompt is identical in both (src/strands_evals/evaluators/evaluator.py): it returns "" when session_history[-1] is a list.

_extract_trace_level did change in #336, but only in how owned_tools is selected — the previous_turns.append(tool_executions) still happens before TraceLevelInput(...) is constructed, so session_history[-1] is still the tool-execution list for any turn that used a tool.

Earliest affected tag, by walking tags for the introducing commit d40a2b3 (#185): v0.1.15.

Verified by strandly-the-agent — worth a second pair of eyes, I can be wrong.

[yonib05]

@strandly-the-agent can you prepare a bug fix PR with proper testing for this issue?

[strandly-the-agent]

@yonib05 The fix is ready, tested, and independently reviewed — but I can't open the PR: strandly-the-agent has no push access to this repo, no existing fork of evals (unlike harness-sdk/devtools), and my guardrails block creating one. Two ways to unblock:

1. Invite strandly-the-agent as a collaborator — I can accept the invitation myself, push the branch, and open the PR from it; or
2. Have my operator allow fork creation, and I'll open it fork-style like my harness-sdk PRs.

Alternatively any maintainer can apply the single squashed commit below with git am — it's PR-ready as is (conventional-commit message, Fixes #355). Ping me once either path is open and I'll finish the PR.

The change (1 src file + 3 test files, +129/−8): Evaluator._extract_user_prompt now walks session_history backwards past tool-execution entries to the nearest message, and only accepts a UserMessage. Fixes the blank USER QUERY in reference-mode CorrectnessEvaluator and the blank target-turn User: in CoherenceEvaluator.

Why evaluator-side, not extractor-side — and verification evidence

Design: the issue offered two fixes. Reordering _extract_trace_level (building TraceLevelInput before appending tool executions) would drop the current turn's tool calls/results from that turn's own judge context in _format_tool_level_prompt/_format_trace_level_prompt — trading a lost prompt for lost tool context. The walk-back keeps session_history chronological for every consumer. Deliberate behavior change: a history ending in an AssistantMessage now yields "" instead of the assistant's text (that value is used specifically as the user prompt; the extractor never produces that shape).

Gates (all on the final tree, editable install verified pointing at the working tree):

• Full unit suite green, run in chunks: 1,872 passed, 0 failed
• ruff format --check ✅ · ruff check ✅ · mypy -p src ✅ (with otel/langfuse/langchain/opensearch extras)
• Revert-test: with src/ reverted to main, all new behavior tests fail (blank-prompt bug reproduced); with the fix, all pass

Independent review loop (fresh-context reviewers, per my SOP):

• Round 1 — CHANGES REQUESTED: the new coherence test's assertion also matched the # Previous turns section, so it passed even with the bug present. Fixed by anchoring on "# Target turn to evaluate:\nUser: …"; discrimination re-verified by revert-test. Two doc/scope notes addressed or consciously deferred.
• Round 2 — APPROVE: reviewer reproduced the bug cold, brute-forced the walk against an independent reference implementation over 400 generated histories (0 mismatches), confirmed the anchored assertion discriminates, and confirmed no other consumer of session_history[-1] exists.
• Deferred as out of scope (both reviewers agreed): (a) content[0] text extraction assumption — pre-existing, unreachable via TraceExtractor; (b) CoherenceEvaluator's # Previous turns includes the current turn's user message, so the judge now sees the prompt twice — pre-existing on main for non-tool turns too, strictly better than a blank prompt; worth its own issue.

The patch — apply with git am on current main (1f752dc)

From 62d0c62aaf6d1a499648157f3c6af5fa9688e0ff Mon Sep 17 00:00:00 2001
From: strandly-the-agent <strandly-the-agent@users.noreply.github.com>
Date: Thu, 13 Aug 2026 15:00:44 +0000
Subject: [PATCH] fix: extract user prompt past trailing tool executions in
 session history

The trace extractor appends a turn's tool executions to session_history
after the UserMessage, so for any turn that used tools,
session_history[-1] is a list[ToolExecution] and _extract_user_prompt
returned an empty string. Reference-mode CorrectnessEvaluator then asked
the judge to grade against a blank USER QUERY, and CoherenceEvaluator
rendered a blank target-turn user prompt.

Walk session_history backwards past tool-execution entries to the
nearest message instead, and only accept a UserMessage as the prompt:
a history ending in an AssistantMessage yields an empty string rather
than assistant text.

Fixed evaluator-side rather than reordering the extractor's appends:
building TraceLevelInput before appending tool executions would drop
the current turn's tool calls/results from that turn's own judge
context in _format_trace_level_prompt.

Fixes #355
---
 src/strands_evals/evaluators/evaluator.py     | 22 +++---
 .../evaluators/test_coherence_evaluator.py    | 28 ++++++++
 .../evaluators/test_correctness_evaluator.py  | 18 +++++
 .../evaluators/test_evaluator.py              | 69 +++++++++++++++++++
 4 files changed, 129 insertions(+), 8 deletions(-)

diff --git a/src/strands_evals/evaluators/evaluator.py b/src/strands_evals/evaluators/evaluator.py
index 4da8c47..f49e7c2 100644
--- a/src/strands_evals/evaluators/evaluator.py
+++ b/src/strands_evals/evaluators/evaluator.py
@@ -136,7 +136,13 @@ class Evaluator(Generic[InputT, OutputT]):
         return parsed_inputs[-1]
 
     def _extract_user_prompt(self, parsed_input: TraceLevelInput) -> str:
-        """Extract user prompt from last message in session history.
+        """Extract the current turn's user prompt from session history.
+
+        Skips trailing tool-execution entries: when a turn used tools, the trace
+        extractor appends the tool executions after the `UserMessage`, so the user
+        prompt is the last non-list message rather than `session_history[-1]`.
+        Only a `UserMessage` qualifies; a history ending in an `AssistantMessage`
+        yields an empty string rather than assistant text.
 
         Args:
             parsed_input: Trace-level input containing session history
@@ -144,15 +150,15 @@ class Evaluator(Generic[InputT, OutputT]):
         Returns:
             User prompt text, or empty string if not available
         """
-        if not parsed_input.session_history:
+        for message in reversed(parsed_input.session_history):
+            if isinstance(message, list):
+                continue
+            if isinstance(message, UserMessage) and self._has_text_content(message):
+                first_content = message.content[0]
+                if isinstance(first_content, TextContent):
+                    return first_content.text
             return ""
 
-        last_msg = parsed_input.session_history[-1]
-        if not isinstance(last_msg, list) and self._has_text_content(last_msg):
-            first_content = last_msg.content[0]
-            if isinstance(first_content, TextContent):
-                return first_content.text
-
         return ""
 
     def _format_tools(self, tools: list[ToolConfig]) -> str:
diff --git a/tests/strands_evals/evaluators/test_coherence_evaluator.py b/tests/strands_evals/evaluators/test_coherence_evaluator.py
index 5349029..a277721 100644
--- a/tests/strands_evals/evaluators/test_coherence_evaluator.py
+++ b/tests/strands_evals/evaluators/test_coherence_evaluator.py
@@ -10,6 +10,9 @@ from strands_evals.types.trace import (
     EvaluationLevel,
     Session,
     SpanInfo,
+    ToolCall,
+    ToolExecutionSpan,
+    ToolResult,
     Trace,
 )
 
@@ -264,3 +267,28 @@ def test_system_prompt_contains_evaluation_criteria():
 
     # Check for important note
     assert "tool output ALWAYS takes priority" in evaluator.system_prompt
+
+
+def test_format_prompt_target_turn_shows_user_prompt_with_tool_call():
+    """Target turn renders the user prompt even when a tool call precedes the answer (#355)."""
+    now = datetime.now()
+    span_info = SpanInfo(session_id="test-session", start_time=now, end_time=now)
+    agent_span = AgentInvocationSpan(
+        span_info=span_info,
+        user_prompt="What is the weather in Seattle?",
+        agent_response="It is 72F in Seattle.",
+        available_tools=[],
+    )
+    tool_span = ToolExecutionSpan(
+        span_info=span_info,
+        tool_call=ToolCall(name="get_weather", arguments={"city": "Seattle"}, tool_call_id="1"),
+        tool_result=ToolResult(content="72F", tool_call_id="1"),
+    )
+    trace = Trace(spans=[agent_span, tool_span], trace_id="trace1", session_id="test-session")
+    session = Session(traces=[trace], session_id="test-session")
+    evaluation_data = EvaluationData(input="What is the weather in Seattle?", actual_trajectory=session, name="test")
+
+    evaluator = CoherenceEvaluator()
+    formatted_prompt = evaluator._format_prompt(evaluator._get_last_turn(evaluation_data))
+
+    assert "# Target turn to evaluate:\nUser: What is the weather in Seattle?" in formatted_prompt
diff --git a/tests/strands_evals/evaluators/test_correctness_evaluator.py b/tests/strands_evals/evaluators/test_correctness_evaluator.py
index 84a44ef..d1e5d68 100644
--- a/tests/strands_evals/evaluators/test_correctness_evaluator.py
+++ b/tests/strands_evals/evaluators/test_correctness_evaluator.py
@@ -209,3 +209,21 @@ def test_reference_score_mapping(
     assert result[0].score == expected_value
     assert result[0].test_pass == expected_pass
     assert result[0].label == verdict
+
+
+@patch("strands_evals.evaluators.correctness_evaluator.Agent")
+def test_reference_prompt_includes_user_query_with_tool_call(mock_agent_class, evaluation_data_with_reference):
+    """Reference-mode judge prompt contains the user query even when a tool call precedes the answer (#355)."""
+    mock_agent = Mock()
+    mock_result = Mock()
+    mock_result.structured_output = CorrectnessReferenceRating(
+        reasoning="Test", verdict=CorrectnessReferenceScore.CORRECT
+    )
+    mock_agent.return_value = mock_result
+    mock_agent_class.return_value = mock_agent
+    evaluator = CorrectnessEvaluator()
+
+    evaluator.evaluate(evaluation_data_with_reference)
+
+    judge_prompt = mock_agent.call_args.args[0]
+    assert "USER QUERY:\nWhat is 2 + 2?" in judge_prompt
diff --git a/tests/strands_evals/evaluators/test_evaluator.py b/tests/strands_evals/evaluators/test_evaluator.py
index 255ecf6..9d75eda 100644
--- a/tests/strands_evals/evaluators/test_evaluator.py
+++ b/tests/strands_evals/evaluators/test_evaluator.py
@@ -1,3 +1,4 @@
+from datetime import datetime
 from unittest.mock import MagicMock
 
 import pytest
@@ -8,10 +9,15 @@ from strands_evals.evaluators.evaluator import DEFAULT_BEDROCK_MODEL_ID
 from strands_evals.types import EvaluationData, EvaluationOutput
 from strands_evals.types.trace import (
     AssistantMessage,
+    SpanInfo,
     TextContent,
+    ToolCall,
     ToolCallContent,
     ToolConfig,
+    ToolExecution,
+    ToolResult,
     ToolResultContent,
+    TraceLevelInput,
     UserMessage,
 )
 
@@ -411,3 +417,66 @@ class TestFormatTools:
         result = self.evaluator._format_tools(tools)
         expected = "- tool_a: First tool\n- tool_b: Second tool\n  Parameters:\n    - x (number (required)): A number"
         assert result == expected
+
+
+class TestExtractUserPrompt:
+    """Tests for _extract_user_prompt, including trailing tool executions (#355)."""
+
+    def setup_method(self):
+        self.evaluator = SimpleEvaluator()
+        now = datetime.now()
+        self.span_info = SpanInfo(session_id="test-session", start_time=now, end_time=now)
+        self.tool_executions = [
+            ToolExecution(
+                tool_call=ToolCall(name="calculator", arguments={"expression": "2+2"}, tool_call_id="1"),
+                tool_result=ToolResult(content="4", tool_call_id="1"),
+            )
+        ]
+
+    def _trace_input(self, session_history):
+        return TraceLevelInput(
+            span_info=self.span_info,
+            agent_response=TextContent(text="The answer is 4."),
+            session_history=session_history,
+        )
+
+    def test_user_message_last(self):
+        """History ending with the UserMessage returns its text."""
+        parsed = self._trace_input([UserMessage(content=[TextContent(text="What is 2 + 2?")])])
+        assert self.evaluator._extract_user_prompt(parsed) == "What is 2 + 2?"
+
+    def test_skips_trailing_tool_executions(self):
+        """Tool executions appended after the UserMessage do not hide the prompt (#355)."""
+        parsed = self._trace_input([UserMessage(content=[TextContent(text="What is 2 + 2?")]), self.tool_executions])
+        assert self.evaluator._extract_user_prompt(parsed) == "What is 2 + 2?"
+
+    def test_multi_turn_history_returns_current_turn_prompt(self):
+        """Only the current turn's prompt is returned, not one from an earlier turn."""
+        parsed = self._trace_input(
+            [
+                UserMessage(content=[TextContent(text="First question")]),
+                self.tool_executions,
+                AssistantMessage(content=[TextContent(text="First answer")]),
+                UserMessage(content=[TextContent(text="Second question")]),
+                self.tool_executions,
+            ]
+        )
+        assert self.evaluator._extract_user_prompt(parsed) == "Second question"
+
+    def test_empty_history_returns_empty(self):
+        parsed = self._trace_input([])
+        assert self.evaluator._extract_user_prompt(parsed) == ""
+
+    def test_assistant_message_last_returns_empty(self):
+        """An AssistantMessage is not mistaken for the user prompt."""
+        parsed = self._trace_input(
+            [
+                UserMessage(content=[TextContent(text="Question")]),
+                AssistantMessage(content=[TextContent(text="Answer")]),
+            ]
+        )
+        assert self.evaluator._extract_user_prompt(parsed) == ""
+
+    def test_only_tool_executions_returns_empty(self):
+        parsed = self._trace_input([self.tool_executions])
+        assert self.evaluator._extract_user_prompt(parsed) == ""
-- 
2.39.5

Prepared by strandly-the-agent — a human should review before merging.