[Storage] Add support for CRC64 content validation#47001
Open
jalauzon-msft wants to merge 14 commits into
Open
[Storage] Add support for CRC64 content validation#47001jalauzon-msft wants to merge 14 commits into
jalauzon-msft wants to merge 14 commits into
Conversation
jalauzon-msft
requested review from
annatisch,
danieljurek,
scbedd,
vincenttran-msft and
weirongw23-msft
as code owners
Contributor
There was a problem hiding this comment.
Pull request overview
Adds Azure Storage CRC64 content validation support across all data-plane Storage SDKs (blob, queue, file-share, file-datalake). Extends the existing validate_content keyword to accept "auto", "crc64", and "md5" (in addition to the legacy bool), with CRC64 implemented via the azure-storage-extensions C extension and the service's structured-message framing.
Changes:
- New shared
validation.py(option parsing, MD5/CRC64 helpers) andstreams_async.py(AsyncStructuredMessageDecoder) duplicated into each Storage package's_shared/folder. policies.py/policies_async.pyrefactored: extracts_prepare_content_validationand_validate_content_response, addsAsyncContentValidationPolicy, drops the oldStorageContentValidation.get_content_md5static method (callers/tests updated tocalculate_content_md5); download paths wrap the response stream in a structured-message decoder when CRC64 is used.- Public clients (
*_client.py+.pyi) widen thevalidate_contentkeyword type and update docstrings; new parametrizedtest_content_validation[_async].pysuites added for file-share and file-datalake, plus async structured-message decoder tests for blob;GenericTestProxyParametrize{1,2}helpers added todevtools_testutils.storage.
Reviewed changes
Copilot reviewed 76 out of 76 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| sdk/storage/azure-storage-blob/azure/storage/blob/_shared/validation.py (and matching files in queue/file-share/file-datalake) | New shared module: parse validate_content, compute MD5/CRC64. |
| sdk/storage/azure-storage-blob/azure/storage/blob/_shared/streams_async.py (and matching files) | New async structured-message decoder mirroring the sync one. |
| sdk/storage/azure-storage-blob/azure/storage/blob/_shared/policies.py (and matching files) | Replace StorageContentValidation body with helper functions; add CRC64/structured-message upload + download handling. |
| sdk/storage/azure-storage-blob/azure/storage/blob/_shared/policies_async.py (and matching files) | Add AsyncContentValidationPolicy (needs body load for MD5) wired through _prepare_content_validation/_validate_content_response. |
| sdk/storage/azure-storage-blob/azure/storage/blob/_shared/base_client_async.py (and matching files) | Swap sync StorageContentValidation for AsyncContentValidationPolicy in async pipeline. |
| sdk/storage/azure-storage-blob/azure/storage/blob/_shared/request_handlers.py | range_validation switched from string "true" to bool True. |
| sdk/storage/azure-storage-blob/azure/storage/blob/_blob_client.py / .pyi / _container_client.* / aio variants | Widen validate_content keyword type, new docs, parse via parse_validation_option, reject CRC64 + client-side encryption combo. |
| sdk/storage/azure-storage-blob/azure/storage/blob/_blob_client_helpers.py | Threads validate_content through _upload_blob_options/_download_blob_options; uses parse_validation_option in stage/upload page/append. |
| sdk/storage/azure-storage-blob/azure/storage/blob/_upload_helpers.py + aio | validate_content typed as CV_TYPE_PARSED; updated check for "original upload path". |
| sdk/storage/azure-storage-blob/azure/storage/blob/_download.py + aio | Use is_md5_validation for MD5-only paths; track _is_structured_message to gate _download_complete. |
| sdk/storage/azure-storage-file-share/azure/storage/fileshare/_file_client.py / aio + .pyi | Same expansion of validate_content for create_file/upload_file/upload_range/download_file. |
| sdk/storage/azure-storage-file-share/azure/storage/fileshare/_download.py + aio | MD5-only behavior gated through is_md5_validation. |
| sdk/storage/azure-storage-file-datalake/azure/storage/filedatalake/_data_lake_file_client.py / aio + .pyi + _helpers | Same widening for upload_data/append_data/download_file. |
| sdk/storage/azure-storage-*/dev_requirements.txt | Add local azure-storage-extensions dep for tests. |
| sdk/storage/azure-storage-*/assets.json | Bump recorded-test asset tags. |
| sdk/storage/azure-storage-file-share/tests/test_content_validation*.py, file-datalake equivalents, azure-storage-blob/tests/test_streams_async.py | New tests for content-validation modes and async structured-message decoder. |
| sdk/storage/azure-storage-blob/tests/test_{append,block,page}_blob*.py | Switch from removed StorageContentValidation.get_content_md5 to calculate_content_md5. |
| eng/tools/azure-sdk-tools/devtools_testutils/storage/{,aio/}{,async}decorators.py + init.py | Add GenericTestProxyParametrize{1,2} helpers exposed via storage testutils. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This change adds support for transactional content validation using the Storage specific CRC64 algorithm for all upload/download APIs across all packages. It makes use of the existing
validate_contentkeyword by just expanding it forcrc64, andauto.