Skip to content

Make test suite faster again, and/or reclassify some test files as quick/slow/veryslow #477

Description

@fingolfin

We recently added a lot of new tests (good!) but unfortunately some of them are rather slow.

That is as such not a fundamental problem, tests just may require time. But at the same time, the slower a test is, the more inconvenient it becomes to run it "quickly" to try out a change.

To deal with this, we classify tests in the categories quick/slow/veryslow, and either place them into the appropriate subdirectory of tst/working/; or into tst/working/combined and then use meta flow control comments like #@if not IsBound(RECOG_TEST_SUITE) or RECOG_TEST_SUITE = "quick" etc. to select which parts of the .tst file is executed for a given test run.

Together that means, we should

  1. try to get .tst files as quick as possible without compromising their utility (test coverage),
  2. ensure tests are classified right.

Here "test coverage" is not meant strictly in mechanical terms as collected by codecov (which is a very simplistic and crude tool), but really "different situations tested".

Right now, on this run, the CI tests took over 13 minutes (821.284 seconds). The testfiles that take more than 10 seconds are these:

10009 ms (513 ms GC) and 1.86GB allocated for veryslow/MatC6.tst
14619 ms (1049 ms GC) and 2.26GB allocated for quick/bugfix.tst
20083 ms (4688 ms GC) and 2.81GB allocated for combined/Sporadic.tst
22156 ms (509 ms GC) and 1.84GB allocated for slow/MatC3_2.tst
34850 ms (2099 ms GC) and 4.75GB allocated for veryslow/PermSym10Subgroup.tst
40783 ms (26540 ms GC) and 32.4GB allocated for combined/MoreNaming.tst
49818 ms (2604 ms GC) and 8.35GB allocated for slow/presentation.tst
54796 ms (5223 ms GC) and 12.0GB allocated for slow/bugfix.tst
59983 ms (1687 ms GC) and 4.54GB allocated for combined/ClassicalNaturalNaming.tst
61672 ms (4875 ms GC) and 12.2GB allocated for veryslow/ClassicalNatural.tst
62643 ms (4454 ms GC) and 1.14GB allocated for slow/GiantPresentation.tst
69737 ms (14331 ms GC) and 14.7GB allocated for slow/GenericSnAnUnknownDegree.tst
94004 ms (2338 ms GC) and 9.93GB allocated for veryslow/MatrixFiniteDeletedPermutationModule.tst
157446 ms (11102 ms GC) and 17.8GB allocated for slow/verification.tst

As we can see, the slowest test by far is slow/verification.tst -- so it should be in veryslow. And perhaps it can be made faster?

On the other extreme, veryslow/MatC6.tst surely is not veryslow; it is slow or perhaps even quick depending on where we want to draw the line (in the end, these classifications are somewhat arbitrary anyway)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions