fix(email): mail search stops broadening on wrong results, not just on no results

[itomek]

Problem

Ask the flagship for a message whose wording you don't remember and it can still hand you a confident wrong answer. PR #4148 fixed the case where the search comes back empty — the tool now retries progressively broader queries. But it only does that when the result count is zero. A query that returns two or three wrong messages looks like a success, so the ladder never runs, the model answers from those messages, and the user is told about mail they never asked about.

Live on a real Gmail mailbox (1137 messages), "sign off on a contract schedule before the end of the week" failed both runs. The message is there — Re: Fieldstone MSA — counter-signature needed before Friday — and searching counter-signature or Fieldstone returns it instantly, so this is the agent layer, not the mailbox and not the connector.

This is the non-empty-but-wrong half of #4134, which recorded only the zero-result half. #4148 genuinely fixes that half (two of three recollection questions now pass on first try) and was not scoped to this one. Consequence for the gate: #4074's "Find" acceptance criterion is still not satisfied after #4148 merges.

Part of #3814. Related: #4134, #4148, #4074.
Spec: none: searched docs/spec/ and docs/plans/ for a flagship mail-retrieval contract — none governs search behaviour; the acceptance contract is the "Find" criterion in #4074.

Outcome

A recollection question whose wording shares no term with the target message either finds the message or says plainly that it didn't — never a confident answer built from unrelated mail.

Acceptance criteria

• Given "asking me to sign off on a contract schedule before the end of the week" on a mailbox containing Re: Fieldstone MSA — counter-signature needed before Friday, the flagship's answer names that thread — today it answers from 2–3 unrelated messages ~7 months older
• search_email no longer treats any non-empty result as final: for a query whose first rung returns a small, low-relevance set, the returned attempts array has more than one entry (today: exactly one, because the loop breaks on the first non-empty rung)
• A non-empty rung that is still judged unusable is reported as such rather than silently accepted — the payload carries a note the model can act on, in the same shape as today's zero-result note
• Results are deduped per thread: search_email("counter-signature", limit=5) returns 5 distinct threads, not 5 messages from one thread (today: one thread consumes every slot)
• Unit tests assert the above two payload shapes directly (multi-rung attempts for a weak first rung; one hit per thread for a multi-message thread), with the mail backend faked
• Live re-run of the three fix(email): the flagship cannot find mail you describe in your own words #4134 recollection questions on a real mailbox: 3 of 3 return the correct thread (today: 2 of 3)

Scope & expectations

• Where: src/gaia/agents/tools/email_tools.py — search_email (the for candidate in _broadening_ladder(...) loop) and _broadening_ladder
• Out of scope: the Gmail/Outlook connectors and the mail backends — both return the target thread correctly on a short query; the per-turn read budget (fix(email): bound mail reads per turn, not just per message #4111)
• Constraints: a longer ladder alone cannot fix this. Term-dropping has no path from contract schedule to counter-signature — there is no shared term to drop down to. Closing this needs a different mechanism (semantic/synonym expansion, or letting the model reformulate from what it already knows about the thread), not more rungs.

How to verify

CLI/agent surface on a machine with a connected Gmail account and the flagship granted. Run the three #4134 recollection questions and show, for each, the query the agent sent, the attempts array, and the thread it named. All three must name the right thread.

🔍 Technical details

Claim class: reproduced (executed) — relayed. The run below was executed on real hardware by another session; this issue relays its evidence and does not re-derive it. The code citations here were opened and checked against the tree at the tested SHA.

Tested at 572e8b79171ce836a712ee4088c3adb862bb42b0 (head of PR #4148, branch issue-4134), flagship run from source, Acer Swift (Ryzen AI 9 465, Radeon 880M, Ubuntu 24.04), live Gmail, 1137 inbox / 447 unread. Full evidence: #4148 (comment)

Question	Result
"argument over cameras police departments use to watch streets"	broadened to cameras police, 1 hit, correct thread, first try — PASS
"a newsletter about Flock cameras"	Flock cameras newsletter, 1 hit, correct thread, first try — PASS
"sign off on a contract schedule before the end of the week"	FAIL, 2 of 2 runs — target thread never returned

Root cause — src/gaia/agents/tools/email_tools.py:441-447:

for candidate in _broadening_ladder(query) or [query]:
    messages = mixin._email_call("search", candidate, limit=_clamp(limit))
    attempts.append({"query": candidate, "count": len(messages)})
    if messages:
        break

The first rung is the query as sent. Any non-empty result ends the loop, so the ladder is unreachable for every query that matches the wrong mail. Only count == 0 reaches rung two.

Compounding half — in the one run where broadening did fire (0 hits on the full query → 11 hits on contract schedule), it stopped on a non-empty but wrong rung, and the payload's broadened note told the model the match was approximate. The model answered from it anyway.

Third contributor — search_email returns per-message hits with no thread grouping (grep -n thread src/gaia/agents/tools/email_tools.py finds only a docstring mention). Searching counter-signature spent all 5 result slots on one conversation, which makes every broadened rung less useful than its limit suggests.

[github-actions]

Confirmed, and the split from #4134 is the right call — but this one can't be picked up on its own yet: the loop it describes doesn't exist on main, it arrives with #4148, which is still open. Whoever takes it should branch off #4148, not main, or the first three acceptance criteria have nothing to attach to.

Two notes for the implementer:

The thread-dedup criterion is cheap — both mail backends already hand back a thread id for every message, so grouping is a few lines in the tool layer with no connector work. Worth doing first; it makes every rung of the ladder more useful on its own.

The recollection criterion is the real design question. As the issue says, there's no term-dropping path from "contract schedule" to "counter-signature", so it needs a new mechanism. The cheapest one is probably letting the model reformulate and search again when a rung comes back weak — it's already in an agent loop — rather than standing up an embedding index over the mailbox, which drags in the embedder that competes with the chat model for the resident model slot.

@kovtcharov-amd — mechanism choice here is a design call, and it's what blocks the "Find" criterion on #4074.

🔍 Technical details

• On main, search_email (src/gaia/agents/tools/email_tools.py:323) is a single _email_call("search", ...) with no ladder, no attempts, no note. The for candidate in _broadening_ladder(query) loop cited as email_tools.py:441-447 is on issue-4134 (PR fix(email): find mail the user describes in their own words #4148). The diagnosis of that loop is accurate: if messages: break makes rungs 2+ reachable only on count == 0.
• Thread ids are already normalized by both backends — src/gaia/agents/tools/_email/gmail.py:213 (threadId) and _email/graph.py:100 (conversationId), both falling back to the message id. So per-thread dedup is a group-by in search_email plus a docstring line; the payload can keep one representative message per thread and a count.
• For the "weak rung" signal, keeping the shape of today's zero-result note (criterion 3) matters more than the heuristic that produces it — the model already has prompt-side handling for that key, and the fix(email): the flagship cannot find mail you describe in your own words #4134 evidence shows a broadened note alone isn't enough to stop it answering from an approximate match. Whatever judges "unusable" should probably suppress the results rather than annotate them.

[github-actions]

This one can't land as its own PR yet — the code every acceptance criterion points at doesn't exist on main. The _broadening_ladder loop, the attempts array and the zero-result note all ship in #4148, which is still open. A fix has to stack on that branch, not land beside it.

Below is the patch I'd stack. It closes the three mechanical criteria — keep climbing past a weak non-empty rung, report a rung that's non-empty but unusable, and dedupe per thread. It does not close the first and last criteria: naming the Fieldstone MSA thread needs the semantic-expansion-or-reformulation mechanism this issue already calls out in its constraints, which is a design call plus a live-mailbox run, not a deterministic patch.

What a human needs to do: apply this on top of issue-4134 (after #4148 settles, since it rewrites the same function), pick the two thresholds deliberately, add the two unit tests, then do the live 3-of-3 re-run. The thresholds below are the reviewable decisions, not settled answers.

🔍 Technical details

Root cause, as filed — email_tools.py on branch issue-4134: the ladder breaks on if messages:, so only count == 0 ever reaches rung two, and hits are per-message with no thread_id grouping even though both backends already normalise one (_email/gmail.py:213, _email/graph.py:100).

Three changes, all in src/gaia/agents/tools/email_tools.py:

1. Climb until the merged result is wide enough, not until it's non-empty. _MIN_DISTINCT_THREADS = 3 is the threshold to argue about — it's the "small, low-relevance set" of criterion 2 made concrete.
2. Dedupe on thread_id, first-seen wins, so the most-specific rung's hit for a thread is the one kept. Each rung over-fetches (capped * 3) so collapsing a 5-message thread doesn't spend 5 of 5 slots.
3. Flag an unconfirmed result: if no returned message contains any content word from the query the caller actually asked for, say so in note and set unconfirmed: true. This is the contract schedule → 11 wrong hits case — the model currently gets a broadened hedge and answers anyway.

--- a/src/gaia/agents/tools/email_tools.py
+++ b/src/gaia/agents/tools/email_tools.py
@@
 _BROADEN_KEEP = 2
 _BROADEN_SINGLES = 3
 
+# A rung that returns one or two threads has found *something*, not
+# necessarily the message. Keep climbing until the merged result is wide
+# enough to choose from, or the ladder runs out.
+_MIN_DISTINCT_THREADS = 3
+
 _BOOLEAN_OPERATORS = frozenset({"AND", "OR", "NOT"})
@@
+def _content_terms(terms: List[str]) -> List[str]:
+    """The terms that say what a message is about — operators and filler gone."""
+    return [
+        t
+        for t in terms
+        if t.upper() not in _BOOLEAN_OPERATORS
+        and not _is_search_operator(t)
+        and t.strip('"').lower() not in _SEARCH_STOPWORDS
+    ]
+
+
+def _thread_key(message: dict) -> str:
+    """Both backends normalise a thread id; fall back to the message id."""
+    return str(message.get("thread_id") or message.get("id") or "")
+
+
+def _merge_by_thread(existing: List[dict], new: List[dict], limit: int) -> List[dict]:
+    """Add a rung's hits, one message per thread, most-specific rung first."""
+    seen = {_thread_key(m) for m in existing}
+    merged = list(existing)
+    for message in new:
+        key = _thread_key(message)
+        if key and key in seen:
+            continue
+        seen.add(key)
+        merged.append(message)
+        if len(merged) >= limit:
+            break
+    return merged
+
+
+def _mentions_any(message: dict, terms: List[str]) -> bool:
+    """True if the message's own words include a term the user asked for."""
+    haystack = " ".join(
+        [
+            str(message.get("subject") or ""),
+            str(message.get("preview") or ""),
+            " ".join(str(a) for a in (message.get("from") or [])),
+        ]
+    ).lower()
+    return any(t.strip('"').lower() in haystack for t in terms)
+
+
 def _broadening_ladder(query: str) -> List[str]:
@@
     _add(terms)
-    content = [
-        t
-        for t in terms
-        if t.upper() not in _BOOLEAN_OPERATORS
-        and not _is_search_operator(t)
-        and t.strip('"').lower() not in _SEARCH_STOPWORDS
-    ]
+    content = _content_terms(terms)
     if content:
@@ search_email
             try:
-                messages: list = []
-                attempts = []
-                for candidate in _broadening_ladder(query) or [query]:
-                    messages = mixin._email_call(
-                        "search", candidate, limit=_clamp(limit)
-                    )
-                    attempts.append({"query": candidate, "count": len(messages)})
-                    if messages:
-                        break
-                used = attempts[-1]["query"]
+                capped = _clamp(limit)
+                # Over-fetch: one chatty thread must not spend every slot.
+                fetch = min(capped * 3, _MAX_LIMIT)
+                messages: list = []
+                attempts = []
+                for candidate in _broadening_ladder(query) or [query]:
+                    hits = mixin._email_call("search", candidate, limit=fetch)
+                    attempts.append({"query": candidate, "count": len(hits)})
+                    messages = _merge_by_thread(messages, hits, capped)
+                    if len(messages) >= _MIN_DISTINCT_THREADS:
+                        break
+                contributing = [a["query"] for a in attempts if a["count"]]
+                used = contributing[0] if contributing else attempts[-1]["query"]
+                asked_for = _content_terms(_search_terms(query))
+                grounded = any(_mentions_any(m, asked_for) for m in messages)
                 payload = {
                     "success": True,
                     "count": len(messages),
                     "order": "relevance",
                     "query_requested": query,
                     "query_used": used,
-                    "broadened": used.split() != query.split(),
+                    "broadened": (
+                        used.split() != query.split() or len(contributing) > 1
+                    ),
                     "attempts": attempts,
                     "messages": messages,
                 }
                 if not messages:
                     payload["note"] = (
                         "No message matched, including the broadest query "
                         f"tried ('{used}'). Do not search again on your own — "
                         "tell the user nothing matched, and ask them for one "
                         "detail that would appear in the message itself, such "
                         "as the sender, a company name, or an amount."
                     )
+                elif not grounded:
+                    payload["unconfirmed"] = True
+                    payload["note"] = (
+                        f"None of these messages contain any word from "
+                        f"'{query}' — they came back from broader queries "
+                        f"({', '.join(repr(q) for q in contributing)}). Do NOT "
+                        "answer from them. Tell the user you could not find "
+                        "the message they described, and ask for one detail "
+                        "that would appear in the message itself — the "
+                        "sender, a company name, or an amount."
+                    )
                 elif payload["broadened"]:

The two unit tests criterion 5 asks for (fake backend, tests/unit/agents/tools/test_email_tools.py):

• a first rung returning one thread → len(out["attempts"]) > 1, and the later rungs' threads are merged in;
• a mailbox where one thread answers with five messages → len({m["thread_id"] for m in out["messages"]}) == len(out["messages"]).

Not covered by this patch. Criterion 1 and the live 3-of-3 (criterion 6) still fail: contract schedule has no term-dropping path to counter-signature, exactly as the issue says. What the patch does change for that case is the failure mode — the model gets unconfirmed: true and is told to say it didn't find it, instead of a broadened hedge it ignores. Whether that's enough for the #4074 "Find" gate is a live-mailbox question.

Not executed. The diff was written against origin/issue-4134 by reading; no test run, no lint run, no mailbox. Thresholds (_MIN_DISTINCT_THREADS, the capped * 3 over-fetch) are proposals.