Add Google Cloud Storage reader support

[mzient]

Category:

New feature (non-breaking change which adds functionality)

Description:

DALI can read from S3 but not from GCS, so datasets on Google Cloud have to be staged elsewhere first. Add a gs:// backend mirroring the existing S3 layer: FileStream::Open and discover_files gain a gs:// branch, guarded by a new BUILD_GCS/GCS_ENABLED option that auto-disables when google-cloud-cpp is absent.

Three details differ from the S3 path. google::cloud::storage::Client is not documented as safe for concurrent use of a single instance, while copies of it are and share the connection pool, so GCSClientManager hands out copies and GCSFileStream owns one instead of borrowing a shared pointer. There is no InitAPI/ShutdownAPI equivalent to serialize, as the library initializes libcurl lazily. Reads land directly in the caller's buffer via std::istream::read, so no preallocated stream buffer is needed, and ReadRange is right-open unlike the inclusive HTTP byte range.

Download checksum validation is off by default: DALI only issues ranged reads and GCS reports checksums for whole objects, so a per-read CRC32C cannot be validated end-to-end. DALI_GCS_VERIFY_CHECKSUMS re-enables it, DALI_GCS_ENDPOINT_URL overrides the endpoint, and DALI_GCS_ANONYMOUS selects insecure credentials for public buckets and emulators.

Additional information:

Affected modules and functionalities:

Key points relevant for the review:

Tests:

• Existing tests apply
• New tests added
  • Python tests
  • GTests
  • Benchmark
  • Other
• N/A

Checklist

Documentation

• Existing documentation applies
• Documentation updated
  • Docstring
  • Doxygen
  • RST
  • Jupyter
  • Other
• N/A

DALI team only

Requirements

• Implements new requirements
• Affects existing requirements
• N/A

REQ IDs: N/A

JIRA TASK: N/A

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.