[AMD][dsv4] fix 2P1D eval 503 "all circuits open" (readiness canary + faster CB recovery)#2255
Open
inkcherry wants to merge 1 commit into
Open
[AMD][dsv4] fix 2P1D eval 503 "all circuits open" (readiness canary + faster CB recovery)#2255inkcherry wants to merge 1 commit into
inkcherry wants to merge 1 commit into
Conversation
…recovery Run 28696443568 (dsv4 2P1D gsm8k eval, c=500x512) failed: from request SemiAnalysisAI#1 the router returned 503 "No available prefill workers (all circuits open or unhealthy)" for all 1319 requests and never recovered, so lm_eval produced 0 result files and "Verify eval scores" failed. The launch script declared "Router is ready for benchmarking" as soon as the router /readiness endpoint returned 200 -- which only proves the router process is up, not that it can actually reach a prefill worker and complete a generation. The eval therefore fired into a router whose prefill path was not yet serving, tripped both circuits, and with the default cb-timeout-duration-secs=60 the circuits stayed open past the client retry budget (~31s) so nothing ever recovered. Fix (server-stability / readiness, no eval concurrency lowered, breaker still fully enabled): - Add an end-to-end readiness canary: require ONE successful generation THROUGH the router (POST /v1/chat/completions, max_tokens=1) before "ready for benchmarking", so the eval never starts against a non-serving prefill path. - Speed circuit-breaker recovery (--cb-timeout-duration-secs 15, --retry-max-retries 3) so a transient trip re-closes inside the client retry budget instead of losing the whole eval. Thresholds unchanged. Satisfies MOTIVATION_RULES clause 3 (Fix) via the approved levers "health-endpoint responsiveness" and "circuit-breaker recovery tuning"; no anti-cheat shortcut (concurrency stays 500/512, breaker + score check remain enabled).
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
inkcherry
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On the 2P1D dsv4 gsm8k eval at concurrency 500, every request gets
503 "No available prefill workers (all circuits open or unhealthy)"from the first request and never recovers, so the eval produces no results.server_sglang.shdeclares the router ready as soon as/readinessreturns 200, which only means the router process is up — not that it can actually reach a prefill worker and finish a generation. The eval fires 500 concurrent requests into a prefill path that isn't serving yet, trips both prefill circuit breakers, and the default breaker timeout (60s) is longer than the client retry budget (~31s), so the circuits never reclose.Changed
POST /v1/chat/completions,max_tokens=1) instead of just/readiness.--cb-timeout-duration-secs 15 --retry-max-retries 3). Breaker and health checks are otherwise unchanged; concurrency and the eval itself are untouched.Validated
conc-500 and conc-512 gsm8k both complete,
exact_match≈ 0.958, no 503s, circuits stay closed.cc @Duyi-Wang @billishyahao