Repository navigation
Replace default review model cascade - #51
Conversation
![postil-dev[bot]](https://avatars.githubusercontent.com/in/3455612?s=60&v=4){kind=small}
There was a problem hiding this comment.
The diff updates default model cascades and scorer models across config, docs, tests, and CI. One finding: the bench-live workflow exposes all four API key secrets to the new scorer-eval job, which is an opt-in path that runs user-controlled model IDs against real endpoints. This is a security concern that needs human review.
warn
.github/workflows/bench-live.yml:80— Scorer eval job exposes all API key secrets · confidence 0.70 · kind: risk
Model: deepseek/deepseek-v4-pro
There was a problem hiding this comment.
The diff introduces a scorer-eval workflow step that reads a JSON report using require inside a shell bun -e one-liner, which will fail because require is not available in Bun's eval context. Additionally, the scorer-eval report upload is gated on inputs.run_scorer_eval == 'true' while the raw-runs upload always includes bench/.runs/scorer-eval, which may leak scorer-eval artifacts even when the eval did not run.
error
.github/workflows/bench-live.yml:103—requireis not defined in Bun eval context · confidence 0.95 · kind: risk.github/workflows/bench-live.yml:174— Raw-runs artifact always includes scorer-eval directory · confidence 0.70 · kind: risk.github/workflows/bench-live.yml:80— Scorer eval job exposes all API key secrets · confidence 0.70 · kind: risk
Model: deepseek/deepseek-v4-pro
There was a problem hiding this comment.
The workflow will fail when run_scorer_eval is true because the scorer_eval_models input was removed but the env var still references it.
.github/workflows/bench-live.yml:79— Restore scorer_eval_models input or hardcode default · confidence 0.80 · kind: risk
Model: deepseek/deepseek-v4-pro
There was a problem hiding this comment.
Removal of three API key env vars from bench-live workflow may break scorer-eval step if it depends on them.
.github/workflows/bench-live.yml:79— Verify scorer-eval does not require removed API keys · confidence 0.60 · kind: uncertainty
Model: deepseek/deepseek-v4-pro
There was a problem hiding this comment.
The scorer model defaults are swapped, and the bench workflow adds scorer eval support.
info
.github/workflows/bench-live.yml:22— Verify scorer eval workflow integration · confidence 0.05 · kind: humanEscalationbench/src/scorer-eval.ts:22— Verify scorer eval test cases · confidence 0.05 · kind: humanEscalationbench/src/scorer-eval.ts:325— Verify true finding generation · confidence 0.05 · kind: humanEscalationbench/src/scorer-eval.ts:334— Verify false finding generation · confidence 0.05 · kind: humanEscalationbench/src/scorer-eval.ts:381— Verify scorer eval aggregation · confidence 0.05 · kind: humanEscalationbench/src/scorer-eval.ts:395— Verify scorer eval report format · confidence 0.05 · kind: humanEscalationbench/src/scorer-eval.ts:431— Verify scorer eval utility functions · confidence 0.05 · kind: humanEscalationbench/src/scorer-eval.ts:270— Verify scorer proxy implementation · confidence 0.05 · kind: humanEscalationbench/src/scorer-eval.ts:161— Verify scorer eval case execution · confidence 0.05 · kind: humanEscalationbench/src/scorer-eval.ts:97— Verify scorer eval main function · confidence 0.05 · kind: humanEscalation
Model: mistralai/mistral-small-3.2-24b-instruct
There was a problem hiding this comment.
The workflow input scorer_model description claims it overrides the scorer for validation, but it is not used in the scorer-eval step; this is a content policy violation (rule 1).
.github/workflows/bench-live.yml:23— Input description contradicts the workflow logic · confidence 0.45 · kind: uncertainty
Model: deepseek/deepseek-v4-pro
The scorer_model input is only used in the live-models bench step, not in the scorer-eval step. Update the description to clarify this.
081a80d to
18ad516
Compare
18ad516 to
081a80d
Compare
There was a problem hiding this comment.
The scorer eval proxy may hang indefinitely if the upstream API request never completes, because server.close() does not terminate in-flight connections. This could cause the optional bench workflow to hang until the job timeout.
bench/src/scorer-eval.ts:298— Scorer eval proxy can hang on upstream fetch, blocking workflow · confidence 0.70 · kind: risk
Model: deepseek/deepseek-v4-pro
… timeout The scorer-eval HTTP proxy was missing a timeout on the upstream fetch call to OpenRouter. If the upstream hung indefinitely, the entire CLI process would be blocked until the outer execFile 240s timeout, with no opportunity for graceful error handling. Added 180s AbortSignal.timeout to the fetch call, leaving sufficient margin before the outer timeout for the proxy to respond with an error instead of hanging indefinitely.
Releases the new default review model cascade and scorer defaults from #51.
1 participant
Updates the compiled review fallback cascade and scorer default using live benchmark evidence from the existing bench harness. The manual live-bench workflow can also upload raw run artifacts and run scorer calibration checks for future roster validation.