Add rerun_if() to request a rerun from inside a test - #382
LouisDeconinck wants to merge 2 commits into
Conversation
Expose a RerunRequest exception and a rerun_if(condition, reason) helper that discards the current attempt and retries while the test still has reruns left. Combined with only_rerun=[RerunRequest], tests can restrict reruns to exactly the known-bad generated-data case instead of any failure. Closes pytest-dev#58. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
icemac
left a comment
icemac
left a comment
There was a problem hiding this comment.
— Comment created by Claude
Automated review: 6 inline findings.
| ) | ||
|
|
||
|
|
||
| class RerunRequest(Exception): |
There was a problem hiding this comment.
— Comment created by Claude
RerunRequest subclasses Exception, so any except Exception: (or pytest.raises(Exception)) between rerun_if() and the test function swallows it. If a test helper or fixture wraps the call like that, no rerun is requested and the test keeps running with the bad data it was trying to reject. pytest's own skip/fail outcomes subclass BaseException for this reason. Consider deriving from BaseException; only_rerun=[RerunRequest] still matches because of the isinstance check.
| this condition and not on other failures. | ||
| """ | ||
| if condition: | ||
| raise RerunRequest(reason or "rerun requested from test") |
There was a problem hiding this comment.
— Comment created by Claude
rerun_if() only raises a normal exception, so a request can still be turned into a hard failure:
- a global or ini filter like
--only-rerun ConnectionError(oronly_rerun = ...in ini) does not matchRerunRequest: ..., so the test fails without being rerun; - a broad
--rerun-exceptregex can block it too; - so can a marker
condition=that evaluates to false.
This contradicts "request a rerun". A more general fix: have _should_hard_fail_on_error treat RerunRequest as always rerunnable when the test has reruns left. Users would then not need to add only_rerun=[RerunRequest] themselves.
|
|
||
|
|
||
| To request a re-run from *inside* a test -- for example when generated data | ||
| happens to hit a known-bad case, similar to ``hypothesis.assume()`` -- use |
There was a problem hiding this comment.
— Comment created by Claude
The hypothesis.assume() comparison is misleading. Inside a @given test, Hypothesis treats RerunRequest as a failure: it shrinks the input, saves the failing example to its database and replays it first on the next attempt. Each rerun then hits the same input again, so the reruns cannot succeed and only add shrinking time. Please mention this, or drop the analogy.
|
|
||
| @pytest.mark.flaky(reruns=2, only_rerun=[RerunRequest]) | ||
| def test_fail(): | ||
| rerun_if(False) |
There was a problem hiding this comment.
— Comment created by Claude
This test never raises RerunRequest (rerun_if(False)), so it would also pass if RerunRequest never matched only_rerun. The main README example (only_rerun=[RerunRequest] plus rerun_if(True) on the first attempt, which should then rerun) has no test. Neither does the global --reruns path that the docs describe.
| failure. Combine with ``only_rerun=[RerunRequest]`` to rerun *only* on | ||
| this condition and not on other failures. | ||
| """ | ||
| if condition: |
There was a problem hiding this comment.
— Comment created by Claude
Minor: reason or "rerun requested from test" replaces any falsy reason, so an explicit reason="" cannot be passed through. "rerun requested from test" if reason is None else reason matches the None default exactly.
| """Raised by :func:`rerun_if` to discard an attempt and request a rerun.""" | ||
|
|
||
|
|
||
| def rerun_if(condition, reason=None): |
There was a problem hiding this comment.
— Comment created by Claude
Cleanup: the new class and function sit between RERUNS_DELAY_BACKOFF_FACTOR_DESC and ONLY_RERUN_DESC, splitting the block of option description constants. Moving them below the constants or near the other public helpers keeps that block together.
2 participants
Summary
rerun_if(condition, reason=None)and theRerunRequestexception: inside a test,rerun_if(...)aborts the current attempt and retries while the test still has reruns (@pytest.mark.flaky(reruns=N)or--reruns)hypothesis.assume()-style use case from the issue; combining withonly_rerun=[RerunRequest]restricts reruns to exactly the requested condition rather than any failureCloses #58
Test plan
rerun_ifinside a flaky test reruns and passesonly_rerun=[RerunRequest]does not rerun other failures