Skip to content

Use metadata-free GCS bucket references#695

Draft
kmontemayor2-sc wants to merge 1 commit into
mainfrom
kmonte/gcs-utils-use-bucket-references
Draft

Use metadata-free GCS bucket references#695
kmontemayor2-sc wants to merge 1 commit into
mainfrom
kmonte/gcs-utils-use-bucket-references

Conversation

@kmontemayor2-sc

Copy link
Copy Markdown
Collaborator

Summary

Replace object-only storage.Client.get_bucket(...) calls in GcsUtils with storage.Client.bucket(...).

These paths only construct blob references, list object prefixes, or delete objects. They do not read bucket metadata, so Client.bucket() is
behavior-preserving and avoids requiring storage.buckets.get for callers that only have object-scoped IAM roles such as roles/ storage.objectUser.

Changes

  • Updated does_gcs_file_exist, delete_gcs_file_if_exist, count_blobs_in_gcs_path, delete_files, and _delete_files_in_bucket_dir to
    use metadata-free bucket references.
  • Added unit coverage that fails if get_bucket() is used for these object-only helpers.
  • Added integration coverage for real GCS object existence, counting, single-object deletion, batch deletion, and directory deletion.

Tests

  • make unit_test_py PY_TEST_FILES="gcs_test.py": passed
  • make integration_test PY_TEST_FILES="gcs_test.py": passed
  • make type_check: passed
  • uv run ruff check --config pyproject.toml gigl/common/utils/gcs.py tests/unit/utils/gcs_test.py tests/integration/common/gcs_test.py:
    passed
  • uv run ruff format --check --config pyproject.toml gigl/common/utils/gcs.py tests/unit/utils/gcs_test.py tests/integration/common/ gcs_test.py: passed

Note: full make check_format passed Python/C++/Scala phases but failed on unrelated pre-existing Markdown formatting issues outside this
change.

Replace object-only Client.get_bucket() calls with Client.bucket() so GcsUtils does not require storage.buckets.get for blob existence, listing, and deletion helpers.

Client.bucket() is a metadata-free drop-in for these paths because they only construct blob references, list object prefixes, or delete objects; none reads bucket metadata.
@kmontemayor2-sc

Copy link
Copy Markdown
Collaborator Author

/all_test

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

GiGL Automation

@ 01:27:39UTC : 🔄 Python Unit Test started.

@ 02:42:51UTC : ✅ Workflow completed successfully.

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

GiGL Automation

@ 01:27:39UTC : 🔄 C++ Unit Test started.

@ 01:29:25UTC : ✅ Workflow completed successfully.

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

GiGL Automation

@ 01:27:39UTC : 🔄 E2E Test started.

@ 03:05:19UTC : ✅ Workflow completed successfully.

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

GiGL Automation

@ 01:27:42UTC : 🔄 Scala Unit Test started.

@ 01:36:26UTC : ✅ Workflow completed successfully.

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

GiGL Automation

@ 01:27:42UTC : 🔄 Integration Test started.

@ 02:56:38UTC : ✅ Workflow completed successfully.

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

GiGL Automation

@ 01:27:43UTC : 🔄 Lint Test started.

@ 01:35:56UTC : ✅ Workflow completed successfully.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants