[Task] Introduce 1-gpu vs 2-gpu pytest markers

[nv-alaiacano]

Description

We are getting resource constrained on how many multi-GPU tests we can run. To address this, we should split apart which tests require being run on 2 GPUs and which can run on a single.

The approach is:

• Create pytest markers (registering them in pytest.ini to avoid warnings): multigpu and singlegpu
• Create new github action workflows to execute on the 2GPU runners, and use the multigpu test fixture.
• Run the rest of the tests on the 1GPU runners

This will allow us to run the single and multi-gpu tests in parallel, and only use the limited multi-gpu resources for tests that actually require it.

Annotating tests

To run tests only in multi-gpu settings:

@pytest.mark.multigpu
def test_multi():
    assert True

To run tests in both single- and multiple-gpu settings:

@pytest.mark.singlegpu
@pytest.mark.multigpu
def test_both():
    assert True

To run tests in single-gpu settings only, no annotation is needed:

def test_single():
    assert True

Running tests

To execute the tests, use the following pytest commands:

To run multi-gpu tests:

pytest -m "multigpu"

To run single-gpu tests, we need two commands. One will run the unannotated tests (most of them) the other will run the single-gpu setting of the ones marked as both singlegpu and multigpu

pytest -m "not multigpu"
pytest  -m "singlegpu"

Repositories

• Merlin @nv-alaiacano split out multi gpu tests from single gpu #999
• Models @gabrielspmoreira
• NVTabular @karlhigley
• Transformers4Rec @rnyak
• Dataloader @edknv Split gpu ci workflow into single-gpu and multi-gpu dataloader#151