Retry timeout before model cascade fallback - #60
Merged
Conversation
OpenRouter and other providers can experience transient slowness where a request is legitimately processing but takes longer than our individual request timeout. Rather than immediately falling back to a weaker model, retry the same model once more (with backoff) before giving up. Timeout retries are separate from transient-error retries (connection errors, retryable HTTP statuses) which already had retry-with-backoff logic. The timeout retry is scoped to keep reviews inside the worker watchdog: request timeout is 240s with up to one 90s retry attempt, and the entire review model phase is capped at 420s. This reduces unnecessary model fallbacks from <10% successful primary model usage to a much higher rate, while maintaining deterministic timeouts and respecting the shared total LLM budget. Tests cover: successful retry after timeout, exhausted retries leading to fallback, and total budget enforcement.
Allow the 8-argument signature for chat_with_temperature and chat_inner since the parameters are all necessary for the transport and retry logic.
![postil-dev[bot]](https://avatars.githubusercontent.com/in/3455612?s=60&v=4){kind=small}
There was a problem hiding this comment.
Gate: failed
The diff introduces timeout retry logic for LLM requests, but the README.md update does not match the new timeout values in the code.
Update README to reflect new timeout values · severity:
error· confidence 0.25 · kind: uncertainty
Review metadata
- Model:
mistralai/mistral-small-3.2-24b-instruct - Review duration: 181.53 s
- Commit:
56bab76 - Dashboard run: View in Postil
- Tokens: 29797 prompt, 16813 completion
- Scorer:
anthropic/claude-haiku-4.5(1 disagreement(s))
Expand the timeout documentation to explain that the timeout retry feature specifically reduces unnecessary fallback to weaker models when the primary model is slow but working. Include explicit mention of retry behavior at the same model level before cascade.
Merged
morgaesis
added a commit
that referenced
this pull request
Jul 12, 2026
Releases timeout-retry-before-cascade-fallback from #60.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
OpenRouter and other providers can experience transient slowness where a request is legitimately processing but takes longer than our individual request timeout. Rather than immediately falling back to a weaker model, retry the same model once more (with backoff) before giving up.
Timeout retries are separate from transient-error retries (connection errors, retryable HTTP statuses) which already had retry-with-backoff logic. The implementation keeps reviews inside the worker watchdog by setting the request timeout to 240s with up to one 90s retry, and capping the entire review model phase at 420s total.
This addresses the issue where deepseek-v4-pro was only serving ~8% of reviews due to unnecessary fallbacks when the model was actually responding but slowly. Testing confirms successful retry after timeout, proper fallback when retries exhaust, and total budget enforcement.