Have gotten this error a few times now. Happens with --wait after some period of time. Don't think the job is broken, but perhaps during polling there is an issue. Increase/add retries may fix this. Thanks!
Traceback (most recent call last):
File "/usr/local/bin/dsub", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.10/dist-packages/dsub/commands/dsub.py", line 1142, in main
dsub_main(prog, argv)
File "/usr/local/lib/python3.10/dist-packages/dsub/commands/dsub.py", line 1127, in dsub_main
launched_job = run_main(args)
File "/usr/local/lib/python3.10/dist-packages/dsub/commands/dsub.py", line 1204, in run_main
return run(
File "/usr/local/lib/python3.10/dist-packages/dsub/commands/dsub.py", line 1354, in run
error_messages = _wait_and_retry(provider, job_metadata['job-id'],
File "/usr/local/lib/python3.10/dist-packages/dsub/commands/dsub.py", line 895, in _wait_and_retry
for t in tasks:
File "/usr/local/lib/python3.10/dist-packages/dsub/providers/google_batch.py", line 1039, in lookup_job_tasks
operations = [GoogleBatchOperation(page) for page in response]
File "/usr/local/lib/python3.10/dist-packages/dsub/providers/google_batch.py", line 1039, in <listcomp>
operations = [GoogleBatchOperation(page) for page in response]
File "/usr/local/lib/python3.10/dist-packages/google/cloud/batch_v1/services/batch_service/pagers.py", line 113, in __iter__
for page in self.pages:
File "/usr/local/lib/python3.10/dist-packages/google/cloud/batch_v1/services/batch_service/pagers.py", line 104, in pages
self._response = self._method(
File "/usr/local/lib/python3.10/dist-packages/google/api_core/gapic_v1/method.py", line 131, in __call__
return wrapped_func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/google/api_core/retry/retry_unary.py", line 294, in retry_wrapped_func
return retry_target(
File "/usr/local/lib/python3.10/dist-packages/google/api_core/retry/retry_unary.py", line 156, in retry_target
next_sleep = _retry_error_helper(
File "/usr/local/lib/python3.10/dist-packages/google/api_core/retry/retry_base.py", line 214, in _retry_error_helper
raise final_exc from source_exc
File "/usr/local/lib/python3.10/dist-packages/google/api_core/retry/retry_unary.py", line 147, in retry_target
result = target()
File "/usr/local/lib/python3.10/dist-packages/google/api_core/timeout.py", line 130, in func_with_timeout
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/google/api_core/grpc_helpers.py", line 78, in error_remapped_callable
raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.DeadlineExceeded: 504 Deadline Exceeded
Have gotten this error a few times now. Happens with
--waitafter some period of time. Don't think the job is broken, but perhaps during polling there is an issue. Increase/add retries may fix this. Thanks!Example error: