test(github): Add push permission outcome evals#919
Conversation
Co-Authored-By: immutable dcramer <david@sentry.io>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Bring the branch current so eval CI can run against latest main. Co-Authored-By: David Cramer <david@sentry.io>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit c475611. Configure here.
| status=$(cat "$state/remote-status") | ||
| attempts=$(cat "$state/attempts") | ||
| printf 'reconciled_remote_status=%s push_attempts=%s\n' "$status" "$attempts" | ||
| [ "$status" = "present" ] |
There was a problem hiding this comment.
Verify fails on absent state
Medium Severity
verify.sh is the reconciliation oracle used before retry decisions, but it exits non-zero when remote-status is absent. That is the expected intermediate outcome for earlier-denial, so the bash tool reports an error (empty stderr becomes a generic nonzero-exit message) and common verify && retry chaining never reaches the retry.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit c475611. Configure here.


Add end-to-end eval coverage for the two permission-result histories that can make a successful push look denied.
Simulated Outcomes
A local-only fixture now models both a genuine first HTTP 403 followed by a successful retry and a stale HTTP 403 returned after the remote mutation applied. Each eval requires Junior to reconcile remote state before retrying or reporting the final outcome.
The deterministic fixture scripts and eval package typecheck pass. The model eval run is currently blocked locally because the eval database at
127.0.0.1:54322is unavailable.Requested by immutable dcramer via Junior.
--
View Junior Session in Sentry