task timeout when processing large artifacts

[philnewm]

Hello there,

I want to mirror artifacts that are not provided through any kind of public repository and require some custom logic to even get the file url for a download.
So creating and assigning a remote would not work in these cases.
One example is the installer of DaVinci Resolve Studio from https://www.blackmagicdesign.com/support/

To achieve this I started utilizing a pulp file repository on my server and uploading content directly.
The upload is implemented via pulp-glue in the python code that fetches the actual download url.

Here an example how I currently create the file content

relative_path: str = f"{product_name}/{product_version}/{Path(url_base).name}"
print(f"processing '{relative_path}' ({index + 1}/{len(missing_artifacts)})")
pulp_file_ctx.create(
  body={
    "file_url": artifact_url,
    "relative_path": relative_path,
    "repository": self.pulp_repo_ctx.entity["name"],
  }
)

I intentionally want to assign the file_url so pulp handles the download and I avoid having to pre-download.
The logic works just fine with most artifacts until I hit any larger artifacts (up to 9GB). in these cases I get (after some minutes) this kind of timeout error.

Task /pulp/api/v3/tasks/019dc454-0e21-7491-a170-5432a115cea0/ failed: '[PLP0005] Request timed out for https://swr.cloud.blackmagicdesign.com/DaVinciResolve/v20.3/DaVinci_Resolve_Studio_20.3_Windows.zip?verify=1777114847-K03MnZAzq5Tb03TwuOIOjtVJgnxt3oA7GgcLQFkfwBM%3D. Increasing the total_timeout value on the remote might help.'
  File "/home/dev_test/artifact_compare/file_providers/blackmagic/sync.py", line 50, in sync
    body={
          "file_url": artifact_url,
          "relative_path": relative_path,
          "repository": self.pulp_repo_ctx.entity["name"],
        }

      )
  File "/home/dev_test/artifact_compare/core/file_base.py", line 58, in sync_repo
    self.sync()
  File "/home/dev_test/artifact_compare/main.py", line 30, in <module>
    bm_sync.sync_repo()
pulp_glue.common.exceptions.PulpException: Task /pulp/api/v3/tasks/019dc454-0e21-7491-a170-5432a115cea0/ failed: '[PLP0005] Request timed out for https://swr.cloud.blackmagicdesign.com/DaVinciResolve/v20.3/DaVinci_Resolve_Studio_20.3_Windows.zip?verify=1777114847-K03MnZAzq5Tb03TwuOIOjtVJgnxt3oA7GgcLQFkfwBM%3D. Increasing the total_timeout value on the remote might help.'

As this error is from the pulp task I assume its from the server side. I think it usually happens between 10-15 minutes.
I'm not sure what timeout I'm hitting here and how I can adjust it. So far I didn't find anything in the docs that resolved this.

On the client site I set this but it only helped in cases where the client was timing out before the task on the server did.

pulp_ctx = PulpContext(
    api_root="/pulp/",
    api_kwargs={
        "base_url": pulp_url,
        "username": pulp_user_name,
        "password": pulp_password,
    },
    verify_ssl=False,
    timeout=timedelta(1800),
)

So let me know if I just missed a setting and can easily adjust this somewhere or if I got any wrong assumptions in my approach.

[pedro-psb]

Hello @philnewm ,
A configuration was added for this in 3.103, but the published docs API are outdated and don't show it.
You can check the most accurate API for your instance in {BASE_URL}/pulp/api/v3/docs/#tag/Content:-Files/operation/content_file_files_upload.

Screenshot in a local instance

[gerrod3]

@philnewm I would set the timeout to a large value that you expect the downloads to be able to complete within. Note if you set the timeout to 0 it will default to 300 secs. https://github.com/pulp/pulpcore/blob/main/pulpcore/download/factory.py#L136-L138

[pedro-psb]

[PLP0005] Request timed out for (...). Increasing the total_timeout value on the remote might help.

Maybe we can improve that error message. There is no remote in this case, which is confusing.

[philnewm]

Hi @pedro-psb, thanks a lot for pointing me to that documentation page right on the server.

Based on this just tried running the file content creation again but so far I just got errors back, my first try was this:

pulp_file_ctx.create(
  body={
    "file_url": artifact_url,
    "relative_path": relative_path,
    "repository": self.pulp_repo_ctx.entity["name"],
    "downloader_config": {
      "total_timeout": 1800.0
    }
  }
)

And this resulted in this error

pulp_glue.common.exceptions.PulpException: Task /pulp/api/v3/tasks/019dd4cf-158e-7f11-aadb-b0921030fa0c/ failed: '{'downloader_config': {'non_field_errors': [ErrorDetail(string='Invalid data. Expected a dictionary, but got str.', code='invalid')]}}'

So I tried this one next

pulp_file_ctx.create(
  body={
    "file_url": artifact_url,
    "relative_path": relative_path,
    "repository": self.pulp_repo_ctx.entity["name"],
    "downloader_config": json.dumps({"total_timeout": 1800.0})
  }
)

And got another error:

pulp_glue.common.exceptions.ValidationError: Validation failed for 'content_file_files_create':
  application/x-www-form-urlencoded: 'body[downloader_config]' is expected to be a object.
  multipart/form-data: 'body[downloader_config]' is expected to be a object.

So it sounds like some validation on the client site is blocking this kind of format or am I missing something?
I did not dive into the pulp-glue code yet to get a deeper understanding.