Skip to content

Propose skip filter instead of run when retry failed tests #536

@davidebianchi

Description

@davidebianchi

At the moment, when --rerun-fails is set, If some tests fail, the rerun is performed with a filter to only run the failed tests.

I propose to add another retry management, to skip passed tests instead of rerun only failed tests. In this way, it could also work if failfast option is set for example.

To use skip effectively, instead of skip only passed tests, we could add an option to define some group of tests that should be skipped if all of them passes.

My proposal is:

  • add a skip-at-prefix option to test run, which enable skip mode and define the test prefix to skip
  • add a never-skip option, if some tests should always be run because are useful for some test setup, for example in complex e2e testing

skip-at-prefix

Imagine we have this test structure:

--- FAIL: TestSiblings
    --- FAIL: TestSiblings/nested
        --- FAIL: TestSiblings/nested/level_1
    --- PASS TestSiblings/pass_1

If we set --skip-at-prefix=TestSiblings/ option, the retry will have -skip ^TestSiblings/pass_1.
With the previous example, if go test run with failfast option, it will retry without skipping tests, since no test passed.

skip-at-prefix will skip by group, so skip will be at Prefix+/subTestName, for each test at that level inside that group. This could be interesting to avoid skipping test too specifically, but grouping by some logic by the user.

never-skip

Imagine we have this test structure, and always want to exec TestSetup test also if passed because it prepare the test environment in some way.

--- PASS: TestSetup

--- FAIL: TestSiblings
    --- FAIL: TestSiblings/nested
        --- FAIL: TestSiblings/nested/level_1
    --- PASS TestSiblings/pass_1

We could set --never-skip=TestSetup, and at each run it will be not added in the skipped tests.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions