-
Notifications
You must be signed in to change notification settings - Fork 107
Add rerun_if() to request a rerun from inside a test #382
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| Add ``rerun_if()`` helper and ``RerunRequest`` exception to request a re-run | ||
| from inside a test (similar to ``hypothesis.assume()``), e.g. combined with | ||
| ``@pytest.mark.flaky(reruns=N, only_rerun=[RerunRequest])`` to retry only on | ||
| the requested condition. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -79,6 +79,25 @@ def works_with_current_xdist(): | |
| "exponential backoff (delay * factor ** (attempt - 1)). defaults to 1.0, " | ||
| "i.e. a constant delay." | ||
| ) | ||
|
|
||
|
|
||
| class RerunRequest(Exception): | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. — Comment created by Claude
|
||
| """Raised by :func:`rerun_if` to discard an attempt and request a rerun.""" | ||
|
|
||
|
|
||
| def rerun_if(condition, reason=None): | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. — Comment created by Claude Cleanup: the new class and function sit between |
||
| """Request a rerun from inside a test when ``condition`` is true. | ||
|
|
||
| The current attempt is aborted and reported as a failure that can be | ||
| retried if the test still has reruns left (via ``@pytest.mark.flaky`` or | ||
| ``--reruns``). Without a rerun budget it behaves like a regular test | ||
| failure. Combine with ``only_rerun=[RerunRequest]`` to rerun *only* on | ||
| this condition and not on other failures. | ||
| """ | ||
| if condition: | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. — Comment created by Claude Minor: |
||
| raise RerunRequest(reason or "rerun requested from test") | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. — Comment created by Claude
This contradicts "request a rerun". A more general fix: have |
||
|
|
||
|
|
||
| ONLY_RERUN_DESC = ( | ||
| "If passed, only rerun errors matching the regex provided. " | ||
| "Pass this flag multiple times (or list one regex per line in the ini " | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -617,6 +617,55 @@ def test_pass(): | |
| assert_outcomes(result, passed=1, rerun=2) | ||
|
|
||
|
|
||
| def test_rerun_if_requests_rerun_from_inside_test(testdir): | ||
| testdir.makepyfile( | ||
| """ | ||
| import pytest | ||
| from pytest_rerunfailures import rerun_if | ||
|
|
||
| import py | ||
| path = py.path.local(__file__).dirpath().ensure('test.res') | ||
|
|
||
| @pytest.mark.flaky(reruns=1) | ||
| def test_pass(): | ||
| count = int(path.read() or 0) | ||
| path.write(count + 1) | ||
| rerun_if(count == 0, 'first attempt has bad data') | ||
| """ | ||
| ) | ||
| result = testdir.runpytest() | ||
| assert_outcomes(result, passed=1, rerun=1) | ||
|
|
||
|
|
||
| def test_rerun_if_only_rerun_restricts_to_requested_reruns(testdir): | ||
| testdir.makepyfile( | ||
| """ | ||
| import pytest | ||
| from pytest_rerunfailures import RerunRequest, rerun_if | ||
|
|
||
| @pytest.mark.flaky(reruns=2, only_rerun=[RerunRequest]) | ||
| def test_fail(): | ||
| rerun_if(False) | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. — Comment created by Claude This test never raises |
||
| assert False | ||
| """ | ||
| ) | ||
| result = testdir.runpytest() | ||
| assert_outcomes(result, passed=0, failed=1, rerun=0) | ||
|
|
||
|
|
||
| def test_rerun_if_without_rerun_budget_fails_normally(testdir): | ||
| testdir.makepyfile( | ||
| """ | ||
| from pytest_rerunfailures import rerun_if | ||
|
|
||
| def test_fail(): | ||
| rerun_if(True) | ||
| """ | ||
| ) | ||
| result = testdir.runpytest() | ||
| assert_outcomes(result, passed=0, failed=1, rerun=0) | ||
|
|
||
|
|
||
| def test_flaky_marker_with_zero_reruns_disables_rerun(testdir): | ||
| testdir.makepyfile( | ||
| f""" | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
— Comment created by Claude
The
hypothesis.assume()comparison is misleading. Inside a@giventest, Hypothesis treatsRerunRequestas a failure: it shrinks the input, saves the failing example to its database and replays it first on the next attempt. Each rerun then hits the same input again, so the reruns cannot succeed and only add shrinking time. Please mention this, or drop the analogy.