[Storage] Add Agents.md to storage folder #47877
+461
−0
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,145 @@ | ||
| # AGENTS.md - Azure SDK for Python: Storage | ||
|
|
||
| This file provides guidance for AI agents (e.g., GitHub Copilot, MCP servers, or LLM-based assistants) working with the Azure Storage SDK packages in this directory. | ||
|
|
||
| ## Storage SDK Overview | ||
|
|
||
| The `sdk/storage/` directory contains the following modules: | ||
|
|
||
| | Package | Azure Service | | ||
| |---------|---------------| | ||
| | `azure-storage-blob` | Azure Blob Storage — object storage for unstructured data | | ||
| | `azure-storage-queue` | Azure Queue Storage — message queuing service | | ||
| | `azure-storage-file-share` | Azure Files — fully managed SMB/NFS file shares | | ||
| | `azure-storage-file-datalake` | Azure Data Lake Storage Gen2 — hierarchical namespace over Blob | | ||
| | `azure-storage-blob-changefeed` | Azure Blob Storage Change Feed — ordered log of blob changes | | ||
| | `azure-storage-extensions` | Native performance extensions for Storage Python SDKs | | ||
| | `azure-mgmt-storage` | Azure Storage Management — storage account and resource management | | ||
| | `azure-mgmt-storagecache` | Azure HPC Cache / Storage Cache management | | ||
| | `azure-mgmt-storagesync` | Azure File Sync management | | ||
| | `azure-mgmt-storageimportexport` | Azure Import/Export Service management | | ||
|
|
||
| ## Rules for AI Agents | ||
|
|
||
| ### Rule 1: Do Not Edit Generated Code | ||
|
|
||
| The following directories contain auto-generated code produced by the Azure REST API code generator. **Do not modify these files directly.** Changes must be made upstream in the API specification or the autorest/TypeSpec configuration and then regenerated. | ||
|
|
||
| - `azure-storage-blob/azure/storage/blob/_generated/` | ||
| - `azure-storage-queue/azure/storage/queue/_generated/` | ||
| - `azure-storage-file-share/azure/storage/fileshare/_generated/` | ||
| - `azure-storage-file-datalake/azure/storage/filedatalake/_generated/` | ||
| - `azure-mgmt-storage/azure/mgmt/storage/` *(entire package is generated)* | ||
| - `azure-mgmt-storagecache/azure/mgmt/storagecache/` *(entire package is generated)* | ||
| - `azure-mgmt-storagesync/azure/mgmt/storagesync/` *(entire package is generated)* | ||
| - `azure-mgmt-storageimportexport/azure/mgmt/storageimportexport/` *(entire package is generated)* | ||
|
|
||
| Handwritten (patch) code lives alongside `_generated/` in the package root and in `_patch.py` files. Edits to patch files are acceptable. | ||
|
|
||
| ### Rule 2: Preserve API Consistency Across Blobs, Queues, Files, and Data Lake | ||
|
|
||
| The four data-plane storage packages (`blob`, `queue`, `file-share`, `file-datalake`) share a common design language: | ||
|
|
||
| - Each service exposes a **service client** (`BlobServiceClient`, `QueueServiceClient`, `ShareServiceClient`, `DataLakeServiceClient`) that handles account-level operations. | ||
| - Each exposes a **container/share/filesystem client** and an **item client** (blob, queue, file/directory) with matching method signatures where applicable. | ||
| - Sync and async clients share the same public method names; the async variants live in a corresponding `aio/` subpackage. | ||
| - When adding or changing a method on one client, verify whether the equivalent clients in the other packages should receive the same change. | ||
|
|
||
| ### Rule 3: Prefer Existing Patterns Over New Abstractions | ||
|
|
||
| Before introducing a new helper class, utility function, or base class, search for an existing implementation across the storage packages. Common patterns already in use include: | ||
|
|
||
| - Shared authentication helpers in `_shared/` subdirectories. | ||
| - `StorageConfiguration` and pipeline-building utilities reused across packages. | ||
| - `_serialize.py` / `_deserialize.py` modules for request/response transformation. | ||
| - `StorageRetryPolicy` and connection-string parsing in `azure-storage-blob/_shared/`. | ||
|
Member
There was a problem hiding this comment. I like this rule in principle, but the examples are not the best. The first and third ones are decent but the second and fourth aren't really things we change often and do not have many patterns. I would maybe look for other examples of where a lot of our helpers live (e.g. various |
||
|
|
||
| Introduce new abstractions only when an existing pattern genuinely cannot accommodate the requirement. | ||
|
|
||
| ### Rule 4: No Magic Strings | ||
|
|
||
| Azure Storage services are case-sensitive for many identifiers (container names, blob names, metadata keys, SAS parameters). Avoid hardcoding raw string literals for: | ||
|
|
||
| - HTTP header names — use constants from `azure.storage.blob._shared.constants` or equivalent. | ||
| - Service version strings — reference the `X_MS_VERSION` constant (from `azure.storage.blob._shared.constants`) rather than inline strings. | ||
| - SAS permission characters — use the typed permission classes (e.g., `BlobSasPermissions`, `QueueSasPermissions`) instead of raw character strings. | ||
| - Error codes — compare against named constants, not literal strings. | ||
|
Member
There was a problem hiding this comment. Again I like this in principle, but we don't really have shared constants files that we use a lot. |
||
|
|
||
|
Member
There was a problem hiding this comment. I think we should have another rule around typing. Could have copilot write this but in general (psuedo code): ### Rule 5: Follow typing conventions
- All public types must have a Python 3 style type hint. Private types should also include a type hint when possible.
- Follow existing patterns with regard to types. Especially in the use of `Literal` over any dedicated enum types.
- If adding an import, specifically for typing, that is not used in runtime code, add it to the `TYPE_CHECKING` conditional block.
Member
There was a problem hiding this comment. kwargs should be |
||
| ## Storage Service Semantics | ||
|
|
||
| Agents modifying service-specific logic should be aware of the following behavioral characteristics: | ||
|
|
||
| ### Blob Storage | ||
|
|
||
| - **AppendBlob**: Append operations are not idempotent. Retrying an `append_block` call on a transient failure can produce duplicate data. Guards against duplicate appends should use the `appendpos_condition` parameter to assert the expected blob length before writing. | ||
| - **BlockBlob commit flow**: Data is uploaded as one or more staged blocks (`stage_block` / `upload_blob_from_url`), then committed atomically via `commit_block_list`. An uncommitted block list is discarded if not committed within 7 days. The public `upload_blob` method handles this flow automatically for large uploads; do not replicate it manually. | ||
| - **PageBlob**: Aligned to 512-byte pages. Writes must start and end on 512-byte boundaries. | ||
|
|
||
| ### Queue Storage | ||
|
|
||
| - **Visibility timeout**: A dequeued message is hidden from other consumers for the duration of the visibility timeout (default 30 seconds, max 7 days). If processing takes longer than the timeout, extend it with `update_message` before it expires, or the message will reappear and be processed again. | ||
| - **Message encoding**: Messages are **not** encoded by default (`NoEncodePolicy`). Base64 encoding is opt-in via `message_encode_policy` (e.g., `TextBase64EncodePolicy` or `BinaryBase64EncodePolicy`) on the `QueueClient`. Changing the policy on an existing queue that already contains messages can make previously written messages undecodable. | ||
|
|
||
| ### Azure Files (File Share) | ||
|
|
||
| - **Share hierarchy**: The resource hierarchy is Account → Share → Directory (nested) → File. The `create_directory` API does not support recursive creation; each parent directory must be created individually before creating a subdirectory. There is no built-in `mkdir -p` equivalent in the SDK. | ||
| - **Lease semantics**: Files (but not directories) support leases. A lease must be acquired before exclusive write access. Shares support snapshot-level leases for backup scenarios. | ||
|
|
||
| ### Data Lake Storage Gen2 | ||
|
|
||
| - **Hierarchical namespace**: Rename and move operations on directories are atomic when the storage account has hierarchical namespace (HNS) enabled. On non-HNS accounts, the Data Lake client falls back to Blob operations and these operations are not atomic. | ||
| - **Access Control Lists (ACLs)**: POSIX-style ACLs are only enforced on HNS-enabled accounts. `set_access_control` calls on non-HNS accounts will succeed but have no effect on authorization. | ||
|
|
||
| ## Build and Test | ||
|
Member
There was a problem hiding this comment. Overall, I think it will be quite difficult for an agent to run the Python tests, especially in Live mode. Running tests is typically very manual and you would not really want to run them all at once as that would take hours. Not really sure how we want to approach that. |
||
|
|
||
| ### Install a package in development mode | ||
|
|
||
| ```bash | ||
| cd sdk/storage/<package-name> | ||
| pip install -e ".[dev]" | ||
|
Member
There was a problem hiding this comment. This install command won't actually work. You would want this: cd sdk/storage/<package-name>
pip install -r dev_requirements.txt
pip install -e .You would also typically want to do this inside a virtualenv but I'm not really sure how that works with agents and if it would be needed or not. |
||
| ``` | ||
|
|
||
| ### Run tests (playback mode — no live service required) | ||
|
|
||
| ```bash | ||
| cd sdk/storage/<package-name> | ||
| pytest tests/ -v | ||
| ``` | ||
|
|
||
| ### Run tests against a live service or local emulator | ||
|
|
||
| Set the appropriate environment variables for the target package (e.g., `AZURE_STORAGE_ACCOUNT_NAME`, `AZURE_STORAGE_ACCOUNT_KEY` or `AZURE_STORAGE_CONNECTION_STRING`), then: | ||
|
Member
There was a problem hiding this comment. It's just Overall though auth will be quite tricky as there are actaully many different account types with many env vars. See here. Various tests use various accounts. Also, some tests use OAuth in which case they don't use an account key but would need |
||
|
|
||
| ```bash | ||
| cd sdk/storage/<package-name> | ||
| pytest tests/ -v --live | ||
|
Member
There was a problem hiding this comment. I don't think |
||
| ``` | ||
|
|
||
| ### Run tests against the Azurite local emulator (optional) | ||
|
|
||
| > **Optional**: Azurite emulator testing is not required. Most tests can be run in playback mode (see above) without any live service or local emulator. Use this section only if you specifically need to validate behavior against a local emulator. | ||
|
|
||
| Start Azurite before running tests: | ||
|
|
||
| ```bash | ||
| # Install Azurite (requires Node.js) | ||
| npm install -g azurite | ||
|
|
||
| # Start all storage services (Blob on 10000, Queue on 10001, Table on 10002) | ||
| azurite --silent --location /tmp/azurite --debug /tmp/azurite/debug.log | ||
| ``` | ||
|
|
||
| Then point the SDK at the local emulator using the well-known Azurite connection string: | ||
|
|
||
| ```bash | ||
| export AZURE_STORAGE_CONNECTION_STRING="DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;QueueEndpoint=http://127.0.0.1:10001/devstoreaccount1;TableEndpoint=http://127.0.0.1:10002/devstoreaccount1;" | ||
| pytest tests/ -v --live | ||
| ``` | ||
|
|
||
| ### Run linting and static analysis | ||
|
|
||
| ```bash | ||
| cd sdk/storage/<package-name> | ||
| azpysdk pylint . | ||
| azpysdk mypy . | ||
|
Member
There was a problem hiding this comment. We should also add |
||
| ``` | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,105 @@ | ||
| # AGENTS.md - Azure Storage Blob SDK | ||
|
|
||
| This file provides package-specific guidance for AI agents working in `sdk/storage/azure-storage-blob/`. | ||
|
|
||
| For repository-wide guidance, see the root [`AGENTS.md`](../../../AGENTS.md). | ||
| For storage-wide guidance, see [`sdk/storage/AGENTS.md`](../AGENTS.md). | ||
|
|
||
| ## Scope | ||
|
|
||
| This package implements Azure Blob Storage clients and supporting models/utilities: | ||
|
|
||
| - Sync and async clients in `azure/storage/blob/` and `azure/storage/blob/aio/` | ||
| - `BlobServiceClient`, `ContainerClient`, `BlobClient`, and `BlobLeaseClient` | ||
| - SAS generation helpers (`generate_blob_sas`, `generate_container_sas`, etc.) | ||
| - Transfer helpers for upload and download | ||
|
|
||
| ## Directory Structure | ||
|
|
||
| ``` | ||
| azure-storage-blob/ | ||
| ├── azure/storage/blob/ | ||
| │ ├── _generated/ # AUTO-GENERATED — do not edit directly | ||
| │ ├── _shared/ # Shared utilities reused by other storage packages | ||
| │ ├── aio/ # Async client implementations | ||
| │ ├── _blob_client.py # BlobClient (handwritten) | ||
| │ ├── _container_client.py # ContainerClient (handwritten) | ||
| │ ├── _blob_service_client.py # BlobServiceClient (handwritten) | ||
| │ └── _patch.py # Handwritten patches applied over generated code | ||
| └── tests/ | ||
| ├── recordings/ # Recorded test cassettes for playback mode | ||
| └── test_*.py # Test files | ||
| ``` | ||
|
|
||
| ## Rules for AI Agents | ||
|
|
||
| ### Rule 1: Do Not Edit Generated Code | ||
|
|
||
| Do not modify anything under `azure/storage/blob/_generated/`. Changes to generated behavior must be made in the upstream API specification or autorest/TypeSpec configuration, then regenerated. | ||
|
|
||
| Handwritten code lives in the package root and in `_patch.py` files — these are the correct locations for targeted changes. | ||
|
|
||
| ### Rule 2: Keep Sync/Async Public Surface Aligned | ||
|
|
||
| The sync clients in `azure/storage/blob/` and the async clients in `azure/storage/blob/aio/` must expose matching public method names and signatures. When changing a method on one, verify the equivalent in the other. | ||
|
|
||
| ### Rule 3: Preserve Client Layering | ||
|
|
||
| Blob APIs follow a strict resource hierarchy: | ||
|
|
||
| ``` | ||
| BlobServiceClient → ContainerClient → BlobClient | ||
| ``` | ||
|
|
||
| Do not move operations to the wrong client level. Prefer existing shared helpers in `_shared/` over new abstractions. | ||
|
|
||
| ### Rule 4: Use Constants and Typed Helpers | ||
|
|
||
| Avoid magic strings. Use existing constants and typed classes for: | ||
|
|
||
| - HTTP header names — use constants from `azure.storage.blob._shared.constants` | ||
| - Service version strings — reference the `X_MS_VERSION` constant | ||
| - SAS permissions — use typed classes (`BlobSasPermissions`, `ContainerSasPermissions`) | ||
| - Error codes — compare against named constants, not literal strings | ||
|
|
||
| ## Blob-Specific Semantics to Preserve | ||
|
|
||
| - **AppendBlob**: Append operations are **not idempotent**. Retrying `append_block` on a transient failure can produce duplicate data. Use the `appendpos_condition` parameter to assert the expected blob length before writing. | ||
| - **BlockBlob**: Data is staged as blocks (`stage_block`) then committed atomically via `commit_block_list`. An uncommitted block list is discarded after 7 days. The `upload_blob` method handles this automatically for large uploads — do not replicate it manually. | ||
| - **PageBlob**: Aligned to 512-byte pages. All writes must start and end on 512-byte boundaries. | ||
|
|
||
| ## Testing Guidance | ||
|
|
||
| ### Playback tests (default — no live service required) | ||
|
|
||
| ```bash | ||
| cd sdk/storage/azure-storage-blob | ||
| pytest tests/ -v | ||
| ``` | ||
|
|
||
| ### Live tests | ||
|
|
||
| Set `AZURE_STORAGE_ACCOUNT_NAME` and either `AZURE_STORAGE_ACCOUNT_KEY` or `AZURE_STORAGE_CONNECTION_STRING`, then: | ||
|
|
||
| ```bash | ||
| cd sdk/storage/azure-storage-blob | ||
| pytest tests/ -v --live | ||
| ``` | ||
|
|
||
| ### Azurite emulator tests (optional) | ||
|
|
||
| > **Optional**: Most tests run in playback mode without a live service or emulator. Use this only when you specifically need emulator behavior. | ||
|
|
||
| See [`sdk/storage/AGENTS.md`](../AGENTS.md) for Azurite setup instructions. | ||
|
|
||
| ## Validation | ||
|
|
||
| ```bash | ||
| cd sdk/storage/azure-storage-blob | ||
| azpysdk pylint . | ||
| azpysdk mypy . | ||
| ``` | ||
|
|
||
| ## Cross-Package Consistency | ||
|
|
||
| The four data-plane storage packages share a common design language. Before adding or changing a method here, check whether the equivalent clients in `azure-storage-queue`, `azure-storage-file-share`, and `azure-storage-file-datalake` should receive the same change. See [`sdk/storage/AGENTS.md`](../AGENTS.md) for details. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,104 @@ | ||
| # AGENTS.md - Azure Storage Data Lake SDK | ||
|
|
||
| This file provides package-specific guidance for AI agents working in `sdk/storage/azure-storage-file-datalake/`. | ||
|
|
||
| For repository-wide guidance, see the root [`AGENTS.md`](../../../AGENTS.md). | ||
| For storage-wide guidance, see [`sdk/storage/AGENTS.md`](../AGENTS.md). | ||
|
|
||
| ## Scope | ||
|
|
||
| This package implements Azure Data Lake Storage Gen2 clients built on top of Azure Blob Storage: | ||
|
|
||
| - Sync and async clients in `azure/storage/filedatalake/` and `azure/storage/filedatalake/aio/` | ||
| - `DataLakeServiceClient`, `FileSystemClient`, `DataLakeDirectoryClient`, and `DataLakeFileClient` | ||
| - Access control list (ACL) management | ||
| - Path and filesystem operations with hierarchical namespace semantics | ||
|
|
||
| ## Directory Structure | ||
|
|
||
| ``` | ||
| azure-storage-file-datalake/ | ||
| ├── azure/storage/filedatalake/ | ||
| │ ├── _generated/ # AUTO-GENERATED — do not edit directly | ||
| │ ├── _shared/ # Shared utilities (symlinked from azure-storage-blob) | ||
| │ ├── aio/ # Async client implementations | ||
| │ ├── _data_lake_service_client.py # DataLakeServiceClient (handwritten) | ||
| │ ├── _file_system_client.py # FileSystemClient (handwritten) | ||
| │ ├── _data_lake_directory_client.py # DataLakeDirectoryClient (handwritten) | ||
| │ ├── _data_lake_file_client.py # DataLakeFileClient (handwritten) | ||
| │ └── _patch.py # Handwritten patches applied over generated code | ||
| └── tests/ | ||
| ├── recordings/ # Recorded test cassettes for playback mode | ||
| └── test_*.py # Test files | ||
| ``` | ||
|
|
||
| ## Rules for AI Agents | ||
|
|
||
| ### Rule 1: Do Not Edit Generated Code | ||
|
|
||
| Do not modify anything under `azure/storage/filedatalake/_generated/`. Changes to generated behavior must be made in the upstream API specification or autorest/TypeSpec configuration, then regenerated. | ||
|
|
||
| Handwritten code lives in the package root and in `_patch.py` files — these are the correct locations for targeted changes. | ||
|
|
||
| ### Rule 2: Preserve the Blob/Data Lake Relationship | ||
|
|
||
| Data Lake Storage Gen2 is a superset of Blob Storage — when hierarchical namespace (HNS) is not enabled on the account, the Data Lake client falls back to Blob Storage operations. This distinction must be preserved: | ||
|
|
||
| - Do not introduce logic that assumes all accounts are HNS-enabled. | ||
| - Non-atomic fallback behavior on non-HNS accounts is correct and expected. | ||
|
|
||
| ### Rule 3: Keep Sync/Async Public Surface Aligned | ||
|
|
||
| The sync clients in `azure/storage/filedatalake/` and the async clients in `azure/storage/filedatalake/aio/` must expose matching public method names and signatures. When changing a method on one, verify the equivalent in the other. | ||
|
|
||
| ### Rule 4: Use Constants and Typed Helpers | ||
|
|
||
| Avoid magic strings. Use existing constants and typed classes for: | ||
|
|
||
| - HTTP header names — use constants from `azure.storage.filedatalake._shared.constants` | ||
| - Service version strings — reference the `X_MS_VERSION` constant | ||
| - SAS permissions — use typed classes (`FileSystemSasPermissions`, `DataLakeSasPermissions`) | ||
| - Error codes — compare against named constants, not literal strings | ||
|
|
||
| ## Data Lake-Specific Semantics to Preserve | ||
|
|
||
| - **Hierarchical namespace (HNS)**: Rename and move operations on directories are **atomic only on HNS-enabled accounts**. On non-HNS accounts, the client falls back to Blob operations and these operations are not atomic. | ||
| - **Access Control Lists (ACLs)**: POSIX-style ACLs are only enforced on HNS-enabled accounts. `set_access_control` calls on non-HNS accounts will succeed but have no effect on authorization. | ||
| - **Path operations**: Paths (directories and files) are first-class resources. Create, rename, delete, and recursive ACL propagation are path-level operations with specific consistency guarantees on HNS accounts. | ||
| - **Pagination of directory listings**: Directory entries are returned paginated; avoid collecting all pages eagerly in a single call when the directory may be large. | ||
|
|
||
| ## Testing Guidance | ||
|
|
||
| ### Playback tests (default — no live service required) | ||
|
|
||
| ```bash | ||
| cd sdk/storage/azure-storage-file-datalake | ||
| pytest tests/ -v | ||
| ``` | ||
|
|
||
| ### Live tests | ||
|
|
||
| Set `AZURE_STORAGE_ACCOUNT_NAME` and either `AZURE_STORAGE_ACCOUNT_KEY` or `AZURE_STORAGE_CONNECTION_STRING`, then: | ||
|
|
||
| ```bash | ||
| cd sdk/storage/azure-storage-file-datalake | ||
| pytest tests/ -v --live | ||
| ``` | ||
|
|
||
| ### Azurite emulator tests (optional) | ||
|
|
||
| > **Optional**: Most tests run in playback mode without a live service or emulator. Use this only when you specifically need emulator behavior. | ||
| See [`sdk/storage/AGENTS.md`](../AGENTS.md) for Azurite setup instructions. | ||
|
|
||
| ## Validation | ||
|
|
||
| ```bash | ||
| cd sdk/storage/azure-storage-file-datalake | ||
| azpysdk pylint . | ||
| azpysdk mypy . | ||
| ``` | ||
|
|
||
| ## Cross-Package Consistency | ||
|
|
||
| The four data-plane storage packages share a common design language. Before adding or changing a method here, check whether the equivalent clients in `azure-storage-blob`, `azure-storage-queue`, and `azure-storage-file-share` should receive the same change. See [`sdk/storage/AGENTS.md`](../AGENTS.md) for details. |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We could add to this to keep the
_sharedfolders in each package in sync always.