Successful subtest rerun still reports failure under xdist

[janmrow]

This seems to be a follow-up to #315 and #330.

Currently, with pytest built-in subtests, a subtest that fails on the first attempt and passes after a rerun, behaves correctly in a serial run. But still leaves the session failed under xdist:

attempt = 0


def test_subtest_recovers(subtests):
    global attempt
    attempt += 1

    with subtests.test():
        assert attempt >= 2

Without xdist

pytest test_repro.py --reruns 1

1 passed, 1 subtests passed, 1 rerun

Exit code: 0.

With xdist

pytest test_repro.py --reruns 1 -n 1

1 failed, 1 passed, 1 subtests passed, 1 rerun

Exit code: 1.

The failed SubtestReport from the first attempt remains accounted for on the controller. Even though the rerun succeeds. When the subtest fails on both attempts, the serial run reports two failures (the final failed subtest and its parent test), while xdist reports three: those two and the superseded first-attempt subtest report.

It seems that pytest emits and xdist forwards the SubtestReport before pytest-rerunfailures decides to rerun the parent test. The later cleanup does not remove the already forwarded report, or its failure accounting from the controller.

Reproduced on current master (eab650f) with pytest 9.1.1 and pytest-xdist 3.8.0.

If this fits the project's scope, I'd be happy to try to work on a fix.

[icemac]

If this is not a problem in pytest itself, it should be handled here.
So a PR is welcome.