Issue with core limit option in Khiops Python library not working with scikit-learn interface or environment variable

[n-voisine]

Hello,

I have noticed that the option in the Khiops Python library to set a maximum number of cores for fitting and predicting does not seem to work when using the scikit-learn interface (khc.fit(X_train, y_train,max_cores=5)) or when setting the environment variable os.environ["KHIOPS_PROC_NUMBER"] = "5".

This is particularly problematic when running Khiops in parallel or concurrently, for example, when multiple tools or processes are using Khiops simultaneously. The inability to limit core usage can lead to resource contention and degraded performance.

Could you please look into this issue? It would be great to have this core limiting feature work as expected across different usage scenarios.

Thank you.

[popescu-v]

These system parameters are not accessible through the Sklearn API. The Core API should be used to this end. However, the Core API allows to change the per-scenario execution system resources. For global system resources, a dedicated API is needed, or the per-API function behavior should be changed with respect to these system parameters, i.e. to change the global settings rather than per-scenario settings.

This could be tackled after #585 .

Nonetheless, the update of the KHIOPS_PROC_NUMBER environment variable can work, provided that it is set before launching Python (e.g. in the shell, via export KHIOPS_PROC_NUMBER=5). The reason for this is that the KhiopsLocalRunner reads the environment variables from khiops_env only once, at the first call of an API function, when the KhiopsLocalRunner instance is created. Hence, the setting in Python via os.environ can also work, provided that it is done before any call of the Khiops Core API in the current Python process context.

[popescu-v]

To be done after #585

[popescu-v]

n_cores will be added to the sklearn estimators: KhiopsClassifier, KhiopsRegressor, KhiopsEncoder and KhiopsCoclustering:

• n_cores to keep the Sklearn naming conventions;
• passed to the estimators rather than to its methods, in order to:
  • keep coherence with the remainder of the Khiops.sklearn code base;
  • keep the Sklearn conventions, whereby fit, predict etc should only accept keyword arguments that are dependent on the data (X) itself.