Support for Azure storage by tramora · Pull Request #571 · KhiopsML/khiops-python

tramora · 2026-04-10T11:36:53Z

Among all the Azure storages, Khiops supports only (via its specific driver) "Files" and "Blobs" (Binary Large Objects)
The only supported authentication method is currently the AZURE_STORAGE_CONNECTION_STRING embedding the account name and account key

Completes #244

TODO Before Asking for a Review

Rebase your branch to the latest version of main (or main-v10)
Make sure all CI workflows are green
When adding a public feature/fix: Update the Unreleased section of CHANGELOG.md (no date)
Self-Review: Review "Files Changed" tab and fix any problems you find
API Docs (only if there are changes in docstrings, rst files or samples):
- Check the docs build without warning: see the log of the API Docs workflow
- Check that your changes render well in HTML: download the API Docs artifact and open index.html
- If there are any problems it is faster to iterate by building locally the API Docs

tramora · 2026-04-13T08:49:36Z

The automated tests on conda environments currently fail because of an outdate version of the azure-storage-file-share package in the conda-forge channel. This PR conda-forge/azure-storage-file-share-feedstock#9 should fix the issue but it is blocked for the moment.

- Among all the Azure storages, Khiops supports only (via its specific driver) "Files" and "Blobs" (Binary Large Objects) - The only supported authentication method is currently the `AZURE_STORAGE_CONNECTION_STRING` embedding the account name and account key

folmos-at-orange · 2026-04-14T08:24:16Z

khiops/core/internals/filesystems.py

+class AzureStorageResourceMixin:
+    """Azure compatible Storage Resource Mixin
+
+    Among all the Azure storages, Khiops supports only (via its specific driver) :


Remove the space before :.

folmos-at-orange · 2026-04-14T08:24:28Z

khiops/core/internals/filesystems.py

+        - Files
+        - Blobs (Binary Large Objects)
+
+    For shared Files, the URI pattern of a resource is the following :


Remove the space before :.

folmos-at-orange · 2026-04-14T08:27:01Z

khiops/core/internals/filesystems.py

+        """
+        Notes:
+             by default, the sdk will list all the blobs belonging to the container.
+             An extract filter is then required to simulate a directory listing.
+             Moreover, the virtual folders hierarchy must be deleted from the result.
+        """


I think this should be a code comment not a docstring, because it is an implementation detail.

folmos-at-orange · 2026-04-14T08:27:27Z

khiops/core/internals/filesystems.py

+        """
+        Notes:
+             in order to be consistent with the other drivers,
+             this method will overwrite the destination blob if it exists.
+             This is not the default behavior.
+             Moreover, if the virtual folders hierarchy does not exist
+             it is created automatically as it is part of the blob name.
+        """


I think this should be a code comment not a docstring, because it is an implementation detail.

folmos-at-orange · 2026-04-14T08:28:08Z

khiops/core/internals/filesystems.py

+        """
+        Notes:
+             as the virtual folder names are part of the blob name,
+             there is no such test like checking a virtual folder existence.
+             Only a blob existence is checked here.
+        """


Same as below: This is a comment.

folmos-at-orange · 2026-04-14T08:28:21Z

khiops/core/internals/filesystems.py

+        """
+        Notes:
+             as the virtual folder names are part of the blob name,
+             there is no such action like removing a virtual folder.
+             Only a blob removal is done here.
+        """


Same as below: This is a comment.

folmos-at-orange · 2026-04-14T08:43:19Z

tests/test_remote_access.py

+                try:
+                    os.remove("/tmp/dummy")
+                except:  # pylint: disable=broad-exception-caught
+                    pass
                fs.copy_to_local(
-                    f"{proto}://{bucket_name}/khiops-cicd/samples/{file}", "/tmp/dummy"
+                    f"{proto}://{bucket_name_or_storage_prefix}/khiops-cicd/"
+                    f"samples/{file}",
+                    "/tmp/dummy",
                )
+                # We cannot use `unittest.TestCase.assertTrue` in a class method
+                assert os.path.isfile("/tmp/dummy")


I don't get why this new code was added

If it stays I'd refactor it as:

Before the loop create /tmp/khiops-dummy

Save each file to /tmp/khiops-dummy/{file}

Do not use assert as it is not a programming error, I think it should be something like if not os.path.isfile(...): raise RuntimeError

Clean /tmp/khiops-dummy afterwards
This avoids the try/except with pass in the middle.

folmos-at-orange · 2026-04-14T08:50:45Z

tests/test_remote_access.py

-    def should_skip_in_a_conda_env(self):
-        # The S3 driver is now released for conda too.
-        # No need to skip the tests any longer in a conda environment
-        return False
-
    def config_exists(self):


I think these elimination should go in a different commit.

folmos-at-orange · 2026-04-14T08:53:42Z

khiops/core/internals/filesystems.py

+class AzureStorageResourceMixin:
+    """Azure compatible Storage Resource Mixin


I think all these docs should go in the classes AzureStorage*Resource.

In the docstring of this class they should point to those, or it could be duplicated.

Also I think we should add an URL to a relevant Azure storage page.

tramora force-pushed the 244-support-azure-storage branch 3 times, most recently from be97119 to 08f91e2 Compare April 10, 2026 14:34

tramora requested review from folmos-at-orange and popescu-v April 10, 2026 15:00

tramora force-pushed the 244-support-azure-storage branch 4 times, most recently from 17fe956 to 80691c3 Compare April 12, 2026 12:59

Support for Azure storage

701678b

- Among all the Azure storages, Khiops supports only (via its specific driver) "Files" and "Blobs" (Binary Large Objects) - The only supported authentication method is currently the `AZURE_STORAGE_CONNECTION_STRING` embedding the account name and account key

tramora force-pushed the 244-support-azure-storage branch from 80691c3 to 701678b Compare April 13, 2026 16:32

folmos-at-orange requested changes Apr 14, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for Azure storage#571

Support for Azure storage#571
tramora wants to merge 1 commit intomainfrom
244-support-azure-storage

tramora commented Apr 10, 2026 •

edited

Loading

Uh oh!

tramora commented Apr 13, 2026

Uh oh!

folmos-at-orange Apr 14, 2026

Uh oh!

folmos-at-orange Apr 14, 2026

Uh oh!

folmos-at-orange Apr 14, 2026

Uh oh!

folmos-at-orange Apr 14, 2026

Uh oh!

folmos-at-orange Apr 14, 2026

Uh oh!

folmos-at-orange Apr 14, 2026

Uh oh!

folmos-at-orange Apr 14, 2026

Uh oh!

folmos-at-orange Apr 14, 2026

Uh oh!

folmos-at-orange Apr 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		class AzureStorageResourceMixin:
		"""Azure compatible Storage Resource Mixin

Conversation

tramora commented Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TODO Before Asking for a Review

Uh oh!

tramora commented Apr 13, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

tramora commented Apr 10, 2026 •

edited

Loading