diff --git a/sdk/storage/AGENTS.md b/sdk/storage/AGENTS.md new file mode 100644 index 000000000000..8fafc84861b6 --- /dev/null +++ b/sdk/storage/AGENTS.md @@ -0,0 +1,145 @@ +# AGENTS.md - Azure SDK for Python: Storage + +This file provides guidance for AI agents (e.g., GitHub Copilot, MCP servers, or LLM-based assistants) working with the Azure Storage SDK packages in this directory. + +## Storage SDK Overview + +The `sdk/storage/` directory contains the following modules: + +| Package | Azure Service | +|---------|---------------| +| `azure-storage-blob` | Azure Blob Storage — object storage for unstructured data | +| `azure-storage-queue` | Azure Queue Storage — message queuing service | +| `azure-storage-file-share` | Azure Files — fully managed SMB/NFS file shares | +| `azure-storage-file-datalake` | Azure Data Lake Storage Gen2 — hierarchical namespace over Blob | +| `azure-storage-blob-changefeed` | Azure Blob Storage Change Feed — ordered log of blob changes | +| `azure-storage-extensions` | Native performance extensions for Storage Python SDKs | +| `azure-mgmt-storage` | Azure Storage Management — storage account and resource management | +| `azure-mgmt-storagecache` | Azure HPC Cache / Storage Cache management | +| `azure-mgmt-storagesync` | Azure File Sync management | +| `azure-mgmt-storageimportexport` | Azure Import/Export Service management | + +## Rules for AI Agents + +### Rule 1: Do Not Edit Generated Code + +The following directories contain auto-generated code produced by the Azure REST API code generator. **Do not modify these files directly.** Changes must be made upstream in the API specification or the autorest/TypeSpec configuration and then regenerated. + +- `azure-storage-blob/azure/storage/blob/_generated/` +- `azure-storage-queue/azure/storage/queue/_generated/` +- `azure-storage-file-share/azure/storage/fileshare/_generated/` +- `azure-storage-file-datalake/azure/storage/filedatalake/_generated/` +- `azure-mgmt-storage/azure/mgmt/storage/` *(entire package is generated)* +- `azure-mgmt-storagecache/azure/mgmt/storagecache/` *(entire package is generated)* +- `azure-mgmt-storagesync/azure/mgmt/storagesync/` *(entire package is generated)* +- `azure-mgmt-storageimportexport/azure/mgmt/storageimportexport/` *(entire package is generated)* + +Handwritten (patch) code lives alongside `_generated/` in the package root and in `_patch.py` files. Edits to patch files are acceptable. + +### Rule 2: Preserve API Consistency Across Blobs, Queues, Files, and Data Lake + +The four data-plane storage packages (`blob`, `queue`, `file-share`, `file-datalake`) share a common design language: + +- Each service exposes a **service client** (`BlobServiceClient`, `QueueServiceClient`, `ShareServiceClient`, `DataLakeServiceClient`) that handles account-level operations. +- Each exposes a **container/share/filesystem client** and an **item client** (blob, queue, file/directory) with matching method signatures where applicable. +- Sync and async clients share the same public method names; the async variants live in a corresponding `aio/` subpackage. +- When adding or changing a method on one client, verify whether the equivalent clients in the other packages should receive the same change. + +### Rule 3: Prefer Existing Patterns Over New Abstractions + +Before introducing a new helper class, utility function, or base class, search for an existing implementation across the storage packages. Common patterns already in use include: + +- Shared authentication helpers in `_shared/` subdirectories. +- `StorageConfiguration` and pipeline-building utilities reused across packages. +- `_serialize.py` / `_deserialize.py` modules for request/response transformation. +- `StorageRetryPolicy` and connection-string parsing in `azure-storage-blob/_shared/`. + +Introduce new abstractions only when an existing pattern genuinely cannot accommodate the requirement. + +### Rule 4: No Magic Strings + +Azure Storage services are case-sensitive for many identifiers (container names, blob names, metadata keys, SAS parameters). Avoid hardcoding raw string literals for: + +- HTTP header names — use constants from `azure.storage.blob._shared.constants` or equivalent. +- Service version strings — reference the `X_MS_VERSION` constant (from `azure.storage.blob._shared.constants`) rather than inline strings. +- SAS permission characters — use the typed permission classes (e.g., `BlobSasPermissions`, `QueueSasPermissions`) instead of raw character strings. +- Error codes — compare against named constants, not literal strings. + +## Storage Service Semantics + +Agents modifying service-specific logic should be aware of the following behavioral characteristics: + +### Blob Storage + +- **AppendBlob**: Append operations are not idempotent. Retrying an `append_block` call on a transient failure can produce duplicate data. Guards against duplicate appends should use the `appendpos_condition` parameter to assert the expected blob length before writing. +- **BlockBlob commit flow**: Data is uploaded as one or more staged blocks (`stage_block` / `upload_blob_from_url`), then committed atomically via `commit_block_list`. An uncommitted block list is discarded if not committed within 7 days. The public `upload_blob` method handles this flow automatically for large uploads; do not replicate it manually. +- **PageBlob**: Aligned to 512-byte pages. Writes must start and end on 512-byte boundaries. + +### Queue Storage + +- **Visibility timeout**: A dequeued message is hidden from other consumers for the duration of the visibility timeout (default 30 seconds, max 7 days). If processing takes longer than the timeout, extend it with `update_message` before it expires, or the message will reappear and be processed again. +- **Message encoding**: Messages are **not** encoded by default (`NoEncodePolicy`). Base64 encoding is opt-in via `message_encode_policy` (e.g., `TextBase64EncodePolicy` or `BinaryBase64EncodePolicy`) on the `QueueClient`. Changing the policy on an existing queue that already contains messages can make previously written messages undecodable. + +### Azure Files (File Share) + +- **Share hierarchy**: The resource hierarchy is Account → Share → Directory (nested) → File. The `create_directory` API does not support recursive creation; each parent directory must be created individually before creating a subdirectory. There is no built-in `mkdir -p` equivalent in the SDK. +- **Lease semantics**: Files (but not directories) support leases. A lease must be acquired before exclusive write access. Shares support snapshot-level leases for backup scenarios. + +### Data Lake Storage Gen2 + +- **Hierarchical namespace**: Rename and move operations on directories are atomic when the storage account has hierarchical namespace (HNS) enabled. On non-HNS accounts, the Data Lake client falls back to Blob operations and these operations are not atomic. +- **Access Control Lists (ACLs)**: POSIX-style ACLs are only enforced on HNS-enabled accounts. `set_access_control` calls on non-HNS accounts will succeed but have no effect on authorization. + +## Build and Test + +### Install a package in development mode + +```bash +cd sdk/storage/ +pip install -e ".[dev]" +``` + +### Run tests (playback mode — no live service required) + +```bash +cd sdk/storage/ +pytest tests/ -v +``` + +### Run tests against a live service or local emulator + +Set the appropriate environment variables for the target package (e.g., `AZURE_STORAGE_ACCOUNT_NAME`, `AZURE_STORAGE_ACCOUNT_KEY` or `AZURE_STORAGE_CONNECTION_STRING`), then: + +```bash +cd sdk/storage/ +pytest tests/ -v --live +``` + +### Run tests against the Azurite local emulator (optional) + +> **Optional**: Azurite emulator testing is not required. Most tests can be run in playback mode (see above) without any live service or local emulator. Use this section only if you specifically need to validate behavior against a local emulator. + +Start Azurite before running tests: + +```bash +# Install Azurite (requires Node.js) +npm install -g azurite + +# Start all storage services (Blob on 10000, Queue on 10001, Table on 10002) +azurite --silent --location /tmp/azurite --debug /tmp/azurite/debug.log +``` + +Then point the SDK at the local emulator using the well-known Azurite connection string: + +```bash +export AZURE_STORAGE_CONNECTION_STRING="DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;QueueEndpoint=http://127.0.0.1:10001/devstoreaccount1;TableEndpoint=http://127.0.0.1:10002/devstoreaccount1;" +pytest tests/ -v --live +``` + +### Run linting and static analysis + +```bash +cd sdk/storage/ +azpysdk pylint . +azpysdk mypy . +``` diff --git a/sdk/storage/azure-storage-blob/AGENTS.md b/sdk/storage/azure-storage-blob/AGENTS.md new file mode 100644 index 000000000000..48ad87388056 --- /dev/null +++ b/sdk/storage/azure-storage-blob/AGENTS.md @@ -0,0 +1,105 @@ +# AGENTS.md - Azure Storage Blob SDK + +This file provides package-specific guidance for AI agents working in `sdk/storage/azure-storage-blob/`. + +For repository-wide guidance, see the root [`AGENTS.md`](../../../AGENTS.md). +For storage-wide guidance, see [`sdk/storage/AGENTS.md`](../AGENTS.md). + +## Scope + +This package implements Azure Blob Storage clients and supporting models/utilities: + +- Sync and async clients in `azure/storage/blob/` and `azure/storage/blob/aio/` +- `BlobServiceClient`, `ContainerClient`, `BlobClient`, and `BlobLeaseClient` +- SAS generation helpers (`generate_blob_sas`, `generate_container_sas`, etc.) +- Transfer helpers for upload and download + +## Directory Structure + +``` +azure-storage-blob/ +├── azure/storage/blob/ +│ ├── _generated/ # AUTO-GENERATED — do not edit directly +│ ├── _shared/ # Shared utilities reused by other storage packages +│ ├── aio/ # Async client implementations +│ ├── _blob_client.py # BlobClient (handwritten) +│ ├── _container_client.py # ContainerClient (handwritten) +│ ├── _blob_service_client.py # BlobServiceClient (handwritten) +│ └── _patch.py # Handwritten patches applied over generated code +└── tests/ + ├── recordings/ # Recorded test cassettes for playback mode + └── test_*.py # Test files +``` + +## Rules for AI Agents + +### Rule 1: Do Not Edit Generated Code + +Do not modify anything under `azure/storage/blob/_generated/`. Changes to generated behavior must be made in the upstream API specification or autorest/TypeSpec configuration, then regenerated. + +Handwritten code lives in the package root and in `_patch.py` files — these are the correct locations for targeted changes. + +### Rule 2: Keep Sync/Async Public Surface Aligned + +The sync clients in `azure/storage/blob/` and the async clients in `azure/storage/blob/aio/` must expose matching public method names and signatures. When changing a method on one, verify the equivalent in the other. + +### Rule 3: Preserve Client Layering + +Blob APIs follow a strict resource hierarchy: + +``` +BlobServiceClient → ContainerClient → BlobClient +``` + +Do not move operations to the wrong client level. Prefer existing shared helpers in `_shared/` over new abstractions. + +### Rule 4: Use Constants and Typed Helpers + +Avoid magic strings. Use existing constants and typed classes for: + +- HTTP header names — use constants from `azure.storage.blob._shared.constants` +- Service version strings — reference the `X_MS_VERSION` constant +- SAS permissions — use typed classes (`BlobSasPermissions`, `ContainerSasPermissions`) +- Error codes — compare against named constants, not literal strings + +## Blob-Specific Semantics to Preserve + +- **AppendBlob**: Append operations are **not idempotent**. Retrying `append_block` on a transient failure can produce duplicate data. Use the `appendpos_condition` parameter to assert the expected blob length before writing. +- **BlockBlob**: Data is staged as blocks (`stage_block`) then committed atomically via `commit_block_list`. An uncommitted block list is discarded after 7 days. The `upload_blob` method handles this automatically for large uploads — do not replicate it manually. +- **PageBlob**: Aligned to 512-byte pages. All writes must start and end on 512-byte boundaries. + +## Testing Guidance + +### Playback tests (default — no live service required) + +```bash +cd sdk/storage/azure-storage-blob +pytest tests/ -v +``` + +### Live tests + +Set `AZURE_STORAGE_ACCOUNT_NAME` and either `AZURE_STORAGE_ACCOUNT_KEY` or `AZURE_STORAGE_CONNECTION_STRING`, then: + +```bash +cd sdk/storage/azure-storage-blob +pytest tests/ -v --live +``` + +### Azurite emulator tests (optional) + +> **Optional**: Most tests run in playback mode without a live service or emulator. Use this only when you specifically need emulator behavior. + +See [`sdk/storage/AGENTS.md`](../AGENTS.md) for Azurite setup instructions. + +## Validation + +```bash +cd sdk/storage/azure-storage-blob +azpysdk pylint . +azpysdk mypy . +``` + +## Cross-Package Consistency + +The four data-plane storage packages share a common design language. Before adding or changing a method here, check whether the equivalent clients in `azure-storage-queue`, `azure-storage-file-share`, and `azure-storage-file-datalake` should receive the same change. See [`sdk/storage/AGENTS.md`](../AGENTS.md) for details. diff --git a/sdk/storage/azure-storage-file-datalake/AGENTS.md b/sdk/storage/azure-storage-file-datalake/AGENTS.md new file mode 100644 index 000000000000..8ab857d7dc6c --- /dev/null +++ b/sdk/storage/azure-storage-file-datalake/AGENTS.md @@ -0,0 +1,104 @@ +# AGENTS.md - Azure Storage Data Lake SDK + +This file provides package-specific guidance for AI agents working in `sdk/storage/azure-storage-file-datalake/`. + +For repository-wide guidance, see the root [`AGENTS.md`](../../../AGENTS.md). +For storage-wide guidance, see [`sdk/storage/AGENTS.md`](../AGENTS.md). + +## Scope + +This package implements Azure Data Lake Storage Gen2 clients built on top of Azure Blob Storage: + +- Sync and async clients in `azure/storage/filedatalake/` and `azure/storage/filedatalake/aio/` +- `DataLakeServiceClient`, `FileSystemClient`, `DataLakeDirectoryClient`, and `DataLakeFileClient` +- Access control list (ACL) management +- Path and filesystem operations with hierarchical namespace semantics + +## Directory Structure + +``` +azure-storage-file-datalake/ +├── azure/storage/filedatalake/ +│ ├── _generated/ # AUTO-GENERATED — do not edit directly +│ ├── _shared/ # Shared utilities (symlinked from azure-storage-blob) +│ ├── aio/ # Async client implementations +│ ├── _data_lake_service_client.py # DataLakeServiceClient (handwritten) +│ ├── _file_system_client.py # FileSystemClient (handwritten) +│ ├── _data_lake_directory_client.py # DataLakeDirectoryClient (handwritten) +│ ├── _data_lake_file_client.py # DataLakeFileClient (handwritten) +│ └── _patch.py # Handwritten patches applied over generated code +└── tests/ + ├── recordings/ # Recorded test cassettes for playback mode + └── test_*.py # Test files +``` + +## Rules for AI Agents + +### Rule 1: Do Not Edit Generated Code + +Do not modify anything under `azure/storage/filedatalake/_generated/`. Changes to generated behavior must be made in the upstream API specification or autorest/TypeSpec configuration, then regenerated. + +Handwritten code lives in the package root and in `_patch.py` files — these are the correct locations for targeted changes. + +### Rule 2: Preserve the Blob/Data Lake Relationship + +Data Lake Storage Gen2 is a superset of Blob Storage — when hierarchical namespace (HNS) is not enabled on the account, the Data Lake client falls back to Blob Storage operations. This distinction must be preserved: + +- Do not introduce logic that assumes all accounts are HNS-enabled. +- Non-atomic fallback behavior on non-HNS accounts is correct and expected. + +### Rule 3: Keep Sync/Async Public Surface Aligned + +The sync clients in `azure/storage/filedatalake/` and the async clients in `azure/storage/filedatalake/aio/` must expose matching public method names and signatures. When changing a method on one, verify the equivalent in the other. + +### Rule 4: Use Constants and Typed Helpers + +Avoid magic strings. Use existing constants and typed classes for: + +- HTTP header names — use constants from `azure.storage.filedatalake._shared.constants` +- Service version strings — reference the `X_MS_VERSION` constant +- SAS permissions — use typed classes (`FileSystemSasPermissions`, `DataLakeSasPermissions`) +- Error codes — compare against named constants, not literal strings + +## Data Lake-Specific Semantics to Preserve + +- **Hierarchical namespace (HNS)**: Rename and move operations on directories are **atomic only on HNS-enabled accounts**. On non-HNS accounts, the client falls back to Blob operations and these operations are not atomic. +- **Access Control Lists (ACLs)**: POSIX-style ACLs are only enforced on HNS-enabled accounts. `set_access_control` calls on non-HNS accounts will succeed but have no effect on authorization. +- **Path operations**: Paths (directories and files) are first-class resources. Create, rename, delete, and recursive ACL propagation are path-level operations with specific consistency guarantees on HNS accounts. +- **Pagination of directory listings**: Directory entries are returned paginated; avoid collecting all pages eagerly in a single call when the directory may be large. + +## Testing Guidance + +### Playback tests (default — no live service required) + +```bash +cd sdk/storage/azure-storage-file-datalake +pytest tests/ -v +``` + +### Live tests + +Set `AZURE_STORAGE_ACCOUNT_NAME` and either `AZURE_STORAGE_ACCOUNT_KEY` or `AZURE_STORAGE_CONNECTION_STRING`, then: + +```bash +cd sdk/storage/azure-storage-file-datalake +pytest tests/ -v --live +``` + +### Azurite emulator tests (optional) + +> **Optional**: Most tests run in playback mode without a live service or emulator. Use this only when you specifically need emulator behavior. + +See [`sdk/storage/AGENTS.md`](../AGENTS.md) for Azurite setup instructions. + +## Validation + +```bash +cd sdk/storage/azure-storage-file-datalake +azpysdk pylint . +azpysdk mypy . +``` + +## Cross-Package Consistency + +The four data-plane storage packages share a common design language. Before adding or changing a method here, check whether the equivalent clients in `azure-storage-blob`, `azure-storage-queue`, and `azure-storage-file-share` should receive the same change. See [`sdk/storage/AGENTS.md`](../AGENTS.md) for details. diff --git a/sdk/storage/azure-storage-file-share/AGENTS.md b/sdk/storage/azure-storage-file-share/AGENTS.md new file mode 100644 index 000000000000..5664d0043e2b --- /dev/null +++ b/sdk/storage/azure-storage-file-share/AGENTS.md @@ -0,0 +1,107 @@ +# AGENTS.md - Azure Storage File Share SDK + +This file provides package-specific guidance for AI agents working in `sdk/storage/azure-storage-file-share/`. + +For repository-wide guidance, see the root [`AGENTS.md`](../../../AGENTS.md). +For storage-wide guidance, see [`sdk/storage/AGENTS.md`](../AGENTS.md). + +## Scope + +This package implements Azure Files clients for SMB/NFS file shares: + +- Sync and async clients in `azure/storage/fileshare/` and `azure/storage/fileshare/aio/` +- `ShareServiceClient`, `ShareClient`, `ShareDirectoryClient`, and `ShareFileClient` +- SAS generation helpers (`generate_share_sas`, `generate_file_sas`) +- Lease support for files and shares + +## Directory Structure + +``` +azure-storage-file-share/ +├── azure/storage/fileshare/ +│ ├── _generated/ # AUTO-GENERATED — do not edit directly +│ ├── _shared/ # Shared utilities (symlinked from azure-storage-blob) +│ ├── aio/ # Async client implementations +│ ├── _share_client.py # ShareClient (handwritten) +│ ├── _share_service_client.py # ShareServiceClient (handwritten) +│ ├── _directory_client.py # ShareDirectoryClient (handwritten) +│ ├── _file_client.py # ShareFileClient (handwritten) +│ └── _patch.py # Handwritten patches applied over generated code +└── tests/ + ├── recordings/ # Recorded test cassettes for playback mode + └── test_*.py # Test files +``` + +## Rules for AI Agents + +### Rule 1: Do Not Edit Generated Code + +Do not modify anything under `azure/storage/fileshare/_generated/`. Changes to generated behavior must be made in the upstream API specification or autorest/TypeSpec configuration, then regenerated. + +Handwritten code lives in the package root and in `_patch.py` files — these are the correct locations for targeted changes. + +### Rule 2: Preserve Client Hierarchy Semantics + +Azure Files has a strict four-level resource hierarchy: + +``` +ShareServiceClient → ShareClient → ShareDirectoryClient → ShareFileClient +``` + +Do not flatten or blur this hierarchy. Operations belong at the correct client level; account-level operations go on the service client, share-level operations on the share client, and so on. + +### Rule 3: Keep Sync/Async Public Surface Aligned + +The sync clients in `azure/storage/fileshare/` and the async clients in `azure/storage/fileshare/aio/` must expose matching public method names and signatures. When changing a method on one, verify the equivalent in the other. + +### Rule 4: Use Constants and Typed Helpers + +Avoid magic strings. Use existing constants and typed classes for: + +- HTTP header names — use constants from `azure.storage.fileshare._shared.constants` +- Service version strings — reference the `X_MS_VERSION` constant +- SAS permissions — use typed classes (`ShareSasPermissions`, `FileSasPermissions`) +- Error codes — compare against named constants, not literal strings + +## File Share-Specific Semantics to Preserve + +- **Directory creation is non-recursive**: `create_directory` creates a single directory. Each parent directory must already exist before creating a subdirectory. There is no built-in `mkdir -p` equivalent in the SDK. +- **Lease behavior differs by resource type**: Files (but not directories) support leases for exclusive write access. Shares support snapshot-level leases for backup scenarios. Preserve these semantics; do not apply file-style leases to directories or share-level leases to files. +- **Share snapshots**: Snapshots are read-only point-in-time copies of a share. They can be addressed via the `snapshot` parameter on `ShareClient`. Do not conflate snapshot operations with live share operations. +- **SMB vs. NFS**: The service supports both SMB and NFS protocols at the share level. Avoid introducing protocol-specific assumptions in shared code paths. + +## Testing Guidance + +### Playback tests (default — no live service required) + +```bash +cd sdk/storage/azure-storage-file-share +pytest tests/ -v +``` + +### Live tests + +Set `AZURE_STORAGE_ACCOUNT_NAME` and either `AZURE_STORAGE_ACCOUNT_KEY` or `AZURE_STORAGE_CONNECTION_STRING`, then: + +```bash +cd sdk/storage/azure-storage-file-share +pytest tests/ -v --live +``` + +### Azurite emulator tests (optional) + +> **Optional**: Most tests run in playback mode without a live service or emulator. Use this only when you specifically need emulator behavior. + +See [`sdk/storage/AGENTS.md`](../AGENTS.md) for Azurite setup instructions. + +## Validation + +```bash +cd sdk/storage/azure-storage-file-share +azpysdk pylint . +azpysdk mypy . +``` + +## Cross-Package Consistency + +The four data-plane storage packages share a common design language. Before adding or changing a method here, check whether the equivalent clients in `azure-storage-blob`, `azure-storage-queue`, and `azure-storage-file-datalake` should receive the same change. See [`sdk/storage/AGENTS.md`](../AGENTS.md) for details.