Skip to content

chore: Migrate gsutil usage to gcloud storage #13619

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
bhandarivijay-png wants to merge 1 commit into
base: master
Choose a base branch
from
Open

chore: Migrate gsutil usage to gcloud storage #13619

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/vision/instance_segmentation.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -511,8 +511,8 @@
"if not os.path.exists(model_ckpt_path):\n",
" os.mkdir(model_ckpt_path)\n",
"\n",
"!gsutil cp gs://tf_model_garden/vision/mobilenet/v2_1.0_float/ckpt-180648.data-00000-of-00001 './model_ckpt/'\n",
"!gsutil cp gs://tf_model_garden/vision/mobilenet/v2_1.0_float/ckpt-180648.index './model_ckpt/'"
"!gcloud storage cp gs://tf_model_garden/vision/mobilenet/v2_1.0_float/ckpt-180648.data-00000-of-00001 './model_ckpt/'\n",
"!gcloud storage cp gs://tf_model_garden/vision/mobilenet/v2_1.0_float/ckpt-180648.index './model_ckpt/'"
]
},
{
Expand Down
4 changes: 2 additions & 2 deletions docs/vision/semantic_segmentation.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -378,8 +378,8 @@
"if not os.path.exists(model_ckpt_path):\n",
" os.mkdir(model_ckpt_path)\n",
"\n",
"!gsutil cp gs://tf_model_garden/cloud/vision-2.0/deeplab/deeplabv3_mobilenetv2_coco/best_ckpt-63.data-00000-of-00001 './model_ckpt/'\n",
"!gsutil cp gs://tf_model_garden/cloud/vision-2.0/deeplab/deeplabv3_mobilenetv2_coco/best_ckpt-63.index './model_ckpt/'"
"!gcloud storage cp gs://tf_model_garden/cloud/vision-2.0/deeplab/deeplabv3_mobilenetv2_coco/best_ckpt-63.data-00000-of-00001 './model_ckpt/'\n",
"!gcloud storage cp gs://tf_model_garden/cloud/vision-2.0/deeplab/deeplabv3_mobilenetv2_coco/best_ckpt-63.index './model_ckpt/'"
]
},
{
Expand Down
10 changes: 5 additions & 5 deletions .../projects/waste_identification_ml/Triton_TF_Cloud_Deployment/client/inference_pipeline.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,7 @@ def main(_) -> None:

# Copy the images folder from GCP to the present directory.
input_directory = (INPUT_DIRECTORY.value).rstrip("/\\")
command = f"gsutil -m cp -r {input_directory} ."
command = f"gcloud storage cp --recursive {input_directory} ."
subprocess.run(command, shell=True, check=True)

# Create a folder to store the predictions.
Expand Down Expand Up @@ -361,7 +361,7 @@ def main(_) -> None:

# Move the tracking visualization to the output directory.
commands = [
f"gsutil -m cp -r {output_folder} {OUTPUT_DIRECTORY.value}",
f"gcloud storage cp --recursive {output_folder} {OUTPUT_DIRECTORY.value}",
f"rm -r {output_folder}",
]
combined_command_1 = " && ".join(commands)
Expand All @@ -383,7 +383,7 @@ def main(_) -> None:

# Move the cropped objects to the output directory.
commands = [
f"gsutil -m cp -r {cropped_obj_folder} {OUTPUT_DIRECTORY.value}",
f"gcloud storage cp --recursive {cropped_obj_folder} {OUTPUT_DIRECTORY.value}",
f"rm -r {cropped_obj_folder}",
]

Expand Down Expand Up @@ -423,11 +423,11 @@ def main(_) -> None:
# Move the folders to the destination bucket.
commands = [
(
"gsutil -m cp -r"
"gcloud storage cp --recursive"
f" {os.path.basename(input_directory)} {OUTPUT_DIRECTORY.value}"
),
f"rm -r {os.path.basename(input_directory)}",
f"gsutil -m cp -r {prediction_folder} {OUTPUT_DIRECTORY.value}",
f"gcloud storage cp --recursive {prediction_folder} {OUTPUT_DIRECTORY.value}",
f"rm -r {prediction_folder}",
]

Expand Down
12 changes: 6 additions & 6 deletions official/projects/waste_identification_ml/llm_applications/milk_pouch_detection/deploy.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -183,11 +183,11 @@ echo ""
# ---

echo "✅ Step 4: Create GCS Buckets..."
gsutil mb \
-p "${PROJECT_ID}" \
-l "${REGION}" \
-c standard \
-b on "gs://${SOURCE_BUCKET_NAME}" \
gcloud storage buckets create "gs://${SOURCE_BUCKET_NAME}" \
--project="${PROJECT_ID}" \
--location="${REGION}" \
--default-storage-class=standard \
--uniform-bucket-level-access \
|| echo "Source Bucket 'gs://${SOURCE_BUCKET_NAME}' already exists."
echo "GCS Buckets are ready."
echo ""
Expand Down Expand Up @@ -324,5 +324,5 @@ echo ""

echo "🚀🚀🚀 Deployment complete! 🚀🚀🚀"
echo "You can upload files to the source bucket to be processed in the next run:"
echo "gsutil cp your-local-image.jpg gs://${SOURCE_BUCKET_NAME}/"
echo "gcloud storage cp your-local-image.jpg gs://${SOURCE_BUCKET_NAME}/"
echo "Check the results in the bucket's subfolders."
2 changes: 1 addition & 1 deletion ...el_inference_with_tracking/Inference_Detectron2_Pipeline_with_Tracking_experimental.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -165,7 +165,7 @@
"source": [
"# # Connect to GCP bucket if your data is store there and copy them locally.\n",
"# !gcloud init\n",
"# gsutil -m cp -r gs://input ."
"# gcloud storage cp --recursive gs://input ."
]
},
{
Expand Down
12 changes: 6 additions & 6 deletions research/object_detection/g3doc/running_pets.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,9 +105,9 @@ copy the files into your GCS bucket (substituting `${YOUR_GCS_BUCKET}`):

```bash
# From tensorflow/models/research/
gsutil cp pet_faces_train.record-* gs://${YOUR_GCS_BUCKET}/data/
gsutil cp pet_faces_val.record-* gs://${YOUR_GCS_BUCKET}/data/
gsutil cp object_detection/data/pet_label_map.pbtxt gs://${YOUR_GCS_BUCKET}/data/pet_label_map.pbtxt
gcloud storage cp pet_faces_train.record-* gs://${YOUR_GCS_BUCKET}/data/
gcloud storage cp pet_faces_val.record-* gs://${YOUR_GCS_BUCKET}/data/
gcloud storage cp object_detection/data/pet_label_map.pbtxt gs://${YOUR_GCS_BUCKET}/data/pet_label_map.pbtxt
```

Please remember the path where you upload the data to, as we will need this
Expand All @@ -128,7 +128,7 @@ Bucket.
``` bash
wget http://storage.googleapis.com/download.tensorflow.org/models/object_detection/faster_rcnn_resnet101_coco_11_06_2017.tar.gz
tar -xvf faster_rcnn_resnet101_coco_11_06_2017.tar.gz
gsutil cp faster_rcnn_resnet101_coco_11_06_2017/model.ckpt.* gs://${YOUR_GCS_BUCKET}/data/
gcloud storage cp faster_rcnn_resnet101_coco_11_06_2017/model.ckpt.* gs://${YOUR_GCS_BUCKET}/data/
```

Remember the path where you uploaded the model checkpoint to, as we will need it
Expand Down Expand Up @@ -160,7 +160,7 @@ sed -i "s|PATH_TO_BE_CONFIGURED|"gs://${YOUR_GCS_BUCKET}"/data|g" \
object_detection/samples/configs/faster_rcnn_resnet101_pets.config

# Copy edited template to cloud.
gsutil cp object_detection/samples/configs/faster_rcnn_resnet101_pets.config \
gcloud storage cp object_detection/samples/configs/faster_rcnn_resnet101_pets.config \
gs://${YOUR_GCS_BUCKET}/data/faster_rcnn_resnet101_pets.config
```

Expand Down Expand Up @@ -285,7 +285,7 @@ command from `tensorflow/models/research/`:

```bash
# From tensorflow/models/research/
gsutil cp gs://${YOUR_GCS_BUCKET}/model_dir/model.ckpt-${CHECKPOINT_NUMBER}.* .
gcloud storage cp gs://${YOUR_GCS_BUCKET}/model_dir/model.ckpt-${CHECKPOINT_NUMBER}.* .
python object_detection/export_inference_graph.py \
--input_type image_tensor \
--pipeline_config_path object_detection/samples/configs/faster_rcnn_resnet101_pets.config \
Expand Down