You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, the remote CI is not reliable and will result in faliures and sporadic failures that are unrelated for the PR in question - that means the remote CI is not usable for practical purposes. Problems:
it may still be that not all failures are temporarily decorated with xfail and skip properly
I think there is a even race condition in tests for the server, which results in sporadic failures, especially if multiple CI runs happen in parallel (which happens in every PR since there is a test matrix)
I am not sure how to address this best long-term; short-term I would suggest to simply skip all the flaky tests.
Mid-term, I think the server tests need to be decoupled from openml-python SDK tests, i.e., replaced by three things:
mock backends for simple API tests
a local server option that runs on the VM. This can follow the extension pattern introduced in the v1-v2 migration, see [ENH] V1 → V2 API Migration #1575
tests for the server API decoupled from tests from openml-python, possibly on a CRON and controlling for race conditions
Currently, the remote CI is not reliable and will result in faliures and sporadic failures that are unrelated for the PR in question - that means the remote CI is not usable for practical purposes. Problems:
xfailandskipproperlyI am not sure how to address this best long-term; short-term I would suggest to simply skip all the flaky tests.
Mid-term, I think the server tests need to be decoupled from
openml-pythonSDK tests, i.e., replaced by three things:openml-python, possibly on a CRON and controlling for race conditions