Feature: Nightly tests with newer GPUs

[v01dXYZ]

The current GPUs used by tests are old, both for Nvidia (T4) and AMD (V520).

The reason is of course the price of running such tests.

There should be some way to schedule some nightly tests in order to benefit from spot prices.

There are likely cheaper options than AWS/Azure/GCP for this as well.

For AMDGPU, the cheapest providers are Vultr and HotAisle.

[hughperkins]

We only use them for running unit tests. For CUDA, the T4 is sufficient to run all our tests except conditional gpu graph. I intend to add a runner just for conditional gpu graph in a few weeks.

For AMD, are there any specific unit tests that you are noticing being skipped on the existing AMD hardware?

For performance tests, we use other GPUs, in Genesis, see for example https://github.com/Genesis-Embodied-AI/Genesis/runs/69297561636

Note that CI tests are just to verify work that has already been tested and benchmarked locally. They are not for use for development work itself.

[v01dXYZ]

If it is only for unit tests, that explains a lot. So, no I don't see any unit tests that are being skipped on AMD hardware AFAIK.

On the other hand, what is the way to check a new optimization bring new benefits for some specific hardware? Is it in the scope of quadrants (do you focus on higher level optimisation and trust NVPTX or LLVM to do the lowering right)? Is there a procedure to test perf degradation?

when you mean tested and benchmarked locally, you mean tested/benchmarked on CPU or a local GPU?

[hughperkins]

De facto we run the Genesis benchmarks as above.

They are defined by this yml file https://github.com/Genesis-Embodied-AI/Genesis/blob/2eae9bc950d64328cdcb5c0143e39e8002f34d8b/.github/workflows/production.yml#L112-L216

Or by the test file itself https://github.com/Genesis-Embodied-AI/Genesis/blob/2eae9bc950d64328cdcb5c0143e39e8002f34d8b/tests/test_rigid_benchmarks.py#L897-L942

[hughperkins]

when you mean tested and benchmarked locally, you mean tested/benchmarked on CPU or a local GPU?

Yes. On a local GPU.

[v01dXYZ]

Thanks for the pointers. So the test suite is genesis itself (which is completely logical). I've got to find a way to spin up DigitalOcean/HotAisle MI300 instances while minimizing costs as much as possible.