Skip to content

fix(gcp): remove unsupported --include flags from gcloud storage rsync #799

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
arniechops wants to merge 14 commits into
base: main
Choose a base branch
from
+8 −8
Open

fix(gcp): remove unsupported --include flags from gcloud storage rsync #799

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
5a3d86f
feat(gcp): add GCS checkpoint path support for model weight pulling
arniechops Apr 2, 2026
e8a4f5f
greptile fixes
arniechops Apr 2, 2026
5626f4a
tests
arniechops Apr 2, 2026
63f02dc
Merge branch 'main' into arnavchopra/gcp-weight-pulling
arniechops Apr 2, 2026
19f70d8
lint fixes
arniechops Apr 2, 2026
6c6df3e
fix: black formatting with line-length=100
arniechops Apr 2, 2026
eab9b60
fix(gcp): use gcloud storage rsync instead of cp for --include support
arniechops Apr 3, 2026
0b5bde3
lint
arniechops Apr 3, 2026
f3025a6
fix: black 24.8.0 formatting
arniechops Apr 3, 2026
8bc986b
fix: isort formatting
arniechops Apr 3, 2026
d9f8d76
fix(gcp): remove unsupported --include flags from gcloud storage rsync
arniechops Apr 3, 2026
99457bd
Merge branch 'main' into arnavchopra/fix-gcs-rsync-include-flags
arniechops Apr 3, 2026
ed69062
fix(gcp): exclude .py files from gcloud rsync when trust_remote_code …
arniechops Apr 3, 2026
afcb6de
fix(tests): update GCS unit tests to match --exclude-only rsync flags
arniechops Apr 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions model-engine/model_engine_server/domain/use_cases/llm_model_endpoint_use_cases.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -730,9 +730,11 @@ def load_model_weights_sub_commands_gcs(
" && /opt/google-cloud-sdk/bin/gcloud config set disable_usage_reporting true 2>/dev/null"
)

file_selection_str = '--include="*.model" --include="*.model.v*" --include="*.json" --include="*.safetensors" --include="*.txt" --exclude="optimizer*"'
if trust_remote_code:
file_selection_str += ' --include="*.py"'
# gcloud storage rsync only supports --exclude (Python regex), not --include.
excludes = ['--exclude="optimizer.*"']
if not trust_remote_code:
excludes.append('--exclude=".*\\.py$"')
file_selection_str = " ".join(excludes)

subcommands.append(
f"/opt/google-cloud-sdk/bin/gcloud storage rsync -r"
Expand Down
8 changes: 3 additions & 5 deletions model-engine/tests/unit/domain/test_llm_use_cases.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -666,8 +666,7 @@ def test_load_model_weights_sub_commands(
" | tar -xz -C /opt"
" && /opt/google-cloud-sdk/bin/gcloud config set disable_usage_reporting true 2>/dev/null",
"/opt/google-cloud-sdk/bin/gcloud storage rsync -r"
' --include="*.model" --include="*.model.v*" --include="*.json"'
' --include="*.safetensors" --include="*.txt" --exclude="optimizer*"'
' --exclude="optimizer.*" --exclude=".*\\.py$"'
" gs://fake-bucket/fake-checkpoint test_folder",
]
assert expected_result == subcommands
Expand All @@ -686,9 +685,8 @@ def test_load_model_weights_sub_commands(
" | tar -xz -C /opt"
" && /opt/google-cloud-sdk/bin/gcloud config set disable_usage_reporting true 2>/dev/null",
"/opt/google-cloud-sdk/bin/gcloud storage rsync -r"
' --include="*.model" --include="*.model.v*" --include="*.json"'
' --include="*.safetensors" --include="*.txt" --exclude="optimizer*"'
' --include="*.py" gs://fake-bucket/fake-checkpoint test_folder',
' --exclude="optimizer.*"'
" gs://fake-bucket/fake-checkpoint test_folder",
]
assert expected_result == subcommands

Expand Down