Skip to content

feat: terminate model output loops with split content/reasoning thresholds - #994

Open
sofwanwork wants to merge 1 commit into
chenyme:mainfrom
sofwanwork:feat/split-doom-loop-thresholds
Open

feat: terminate model output loops with split content/reasoning thresholds#994
sofwanwork wants to merge 1 commit into
chenyme:mainfrom
sofwanwork:feat/split-doom-loop-thresholds

Conversation

@sofwanwork

Copy link
Copy Markdown

Why

A looping upstream stream currently runs until the response budget is exhausted. There is no repetition guard in ConvertResponseStream, so a model that gets stuck emitting the same delta burns account quota and fills the client context window before anything stops it.

The naive fix — one shared repeat counter — is worse than no fix for thinking models. high/xhigh effort reasoning legitimately repeats the same short marker ("so", "hmm", "wait", list bullets) many times in a row while thinking. A single counter tuned low enough to catch a real content loop truncates valid deep-thinking responses instead.

What changed

Consecutive identical deltas are tracked per channel, and the stream terminates once a channel exceeds its own ceiling:

channel ceiling rationale
visible content (textDelta) 128 a real content loop burns quota, so this stays well below the reasoning ceiling
reasoning (reasoningSummaryDelta, emitReasoningDelta) 256 deep thinking reuses short markers far more often than visible output does

Two details worth reviewing:

  • Reasoning shares one counter across both paths. reasoningSummaryDelta (buffered summaries) and emitReasoningDelta (raw reasoning) both route through trackReasoningRepeat, so a loop that straddles summary and raw events is still caught. The error messages stay distinct for diagnosis.
  • The content ceiling has to clear legitimate visible repetition. Markdown horizontal rules and ASCII table borders stream as long runs of an identical single-character delta ("-", "=", "|"), and wide tables with empty cells repeat the same separator delta. A ceiling near typical rule width would kill correct output, which is why 128 rather than something tighter.

Counters reset whenever the delta changes, so alternating or interleaved output cannot accumulate into a false positive. Empty deltas are ignored.

Tests

backend/internal/infra/provider/conversation/stream_doomloop_test.go covers both directions:

  • a visible-content loop terminates with model output loop detected
  • a reasoning run between the two ceilings survives and still delivers the visible answer (both reasoning event types)
  • a runaway reasoning loop past its own ceiling still terminates — the higher ceiling is a ceiling, not an exemption
  • markdown rules and table borders (80 identical deltas) complete normally
  • alternating content deltas and reasoning interleaved with distinct content do not trip either counter

The threshold-relative tests derive their repeat counts from the constants and assert the invariant, so they stay meaningful if the ceilings are retuned.

Verified on this branch: go build ./..., go vet, and the full go test ./... suite (62 packages, 0 failures).

…holds

A looping upstream stream burns account quota and the client context window
until the response budget is exhausted. Track consecutive identical deltas
and terminate the stream once a channel exceeds its ceiling.

Content and reasoning keep separate counters. high/xhigh effort reasoning
legitimately repeats the same short marker ("so", "hmm", "wait", list
bullets) far more often than visible output does, so a single shared
counter truncates valid deep-thinking responses. Reasoning is therefore
allowed a higher ceiling (256) than visible content (128).

The content ceiling also has to clear legitimate visible repetition:
markdown horizontal rules and ASCII table borders stream as long runs of
an identical single-character delta ("-", "=", "|").
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant