databricks fs cp fails when volume is empty

[robinstauntoncollins]

Describe the issue

databricks fs cp fails with "Bad Request" when copying a file to a UC managed volume that exists but is empty. The CLI performs a directory existence check via the Files API before writing, and an empty UC volume returns HTTP 404 for that check — causing the CLI to abort instead of writing the file.

Steps to reproduce the behavior

1. Create a UC managed volume: databricks volumes create mycatalog myschema myvol MANAGED
2. Upload a file to it: databricks fs cp myfile.xml dbfs:/Volumes/mycatalog/myschema/myvol/ (succeeds)
3. Delete all files from the volume (e.g. via the Databricks UI)
4. Re-run the same databricks fs cp command

Expected Behavior

The file is uploaded successfully. The volume object still exists; only its contents were deleted. A write operation to an existing volume should not depend on whether the volume currently has any files in it.

Actual Behavior

The command fails with Error: Bad Request. The CLI calls HEAD /api/2.0/fs/directories/Volumes/mycatalog/myschema/myvol before writing. An empty UC volume returns 404 for this endpoint. The CLI interprets this as "no such directory" and aborts without writing the file.

Debug trace:
HEAD /api/2.0/fs/directories/Volumes/mycatalog/myschema/myvol → 404 Not Found
HEAD /api/2.0/fs/files/Volumes/mycatalog/myschema/myvol → 404 Not Found
HEAD /api/2.0/fs/directories/Volumes/mycatalog/myschema → 400 Bad Request
Error: Bad Request

The volume object exists throughout — databricks volumes read mycatalog.myschema.myvol returns full metadata. The --overwrite flag does not help; the same error occurs.

Workaround: Use the Files REST API directly — PUT /api/2.0/fs/files/Volumes/mycatalog/myschema/myvol/myfile.xml — which writes unconditionally without a prior directory check.

OS and CLI version

• CLI: Databricks CLI v1.3.0
• OS: Linux 5.10.16.3 (WSL2 on Windows)

Is this a regression?

Unknown — this was first observed on v1.3.0. Not tested on earlier versions.

Debug Logs

GET /api/2.0/fs/directories/Volumes/mycatalog/myschema/myvol
< HTTP/2 404
HEAD /api/2.0/fs/directories/Volumes/mycatalog/myschema/myvol
< HTTP/2 404
HEAD /api/2.0/fs/files/Volumes/mycatalog/myschema/myvol
< HTTP/2 404
HEAD /api/2.0/fs/directories/Volumes/mycatalog/myschema
< HTTP/2 400
Error: Bad Request

[janniklasrose]

Thank you for the report! Are you still impacted by this, and is this consistently reproducible?

I investigated on my side and I'm unable to reproduce the issue against several environments I have access to (incl AWS and Azure) - getting 200 on the empty volume every time. Could you please share more details about the workspace you are deploying to?