-
Notifications
You must be signed in to change notification settings - Fork 156
Description
At the moment, when --rerun-fails is set, If some tests fail, the rerun is performed with a filter to only run the failed tests.
I propose to add another retry management, to skip passed tests instead of rerun only failed tests. In this way, it could also work if failfast option is set for example.
To use skip effectively, instead of skip only passed tests, we could add an option to define some group of tests that should be skipped if all of them passes.
My proposal is:
- add a
skip-at-prefixoption to test run, which enable skip mode and define the test prefix to skip - add a
never-skipoption, if some tests should always be run because are useful for some test setup, for example in complex e2e testing
skip-at-prefix
Imagine we have this test structure:
--- FAIL: TestSiblings
--- FAIL: TestSiblings/nested
--- FAIL: TestSiblings/nested/level_1
--- PASS TestSiblings/pass_1
If we set --skip-at-prefix=TestSiblings/ option, the retry will have -skip ^TestSiblings/pass_1.
With the previous example, if go test run with failfast option, it will retry without skipping tests, since no test passed.
skip-at-prefix will skip by group, so skip will be at Prefix+/subTestName, for each test at that level inside that group. This could be interesting to avoid skipping test too specifically, but grouping by some logic by the user.
never-skip
Imagine we have this test structure, and always want to exec TestSetup test also if passed because it prepare the test environment in some way.
--- PASS: TestSetup
--- FAIL: TestSiblings
--- FAIL: TestSiblings/nested
--- FAIL: TestSiblings/nested/level_1
--- PASS TestSiblings/pass_1
We could set --never-skip=TestSetup, and at each run it will be not added in the skipped tests.