Skip to content

Added support for UC external locations (direct mode only)#4484

Open
andrewnester wants to merge 9 commits intomainfrom
feat/external-locations
Open

Added support for UC external locations (direct mode only)#4484
andrewnester wants to merge 9 commits intomainfrom
feat/external-locations

Conversation

@andrewnester
Copy link
Contributor

Changes

Added support for UC external locations (direct mode only)

Why

This is a natural extension of support for UC catalogs where UC catalogs can have references to UC external locations.

Tests

Added an acceptance test

@eng-dev-ecosystem-bot
Copy link
Collaborator

eng-dev-ecosystem-bot commented Feb 10, 2026

Commit: 0fd9a36

Run: 21906584594

Env ❌​FAIL 🟨​KNOWN 🔄​flaky 💚​RECOVERED 🙈​SKIP ✅​pass 🙈​skip Time
🟨​ aws linux 7 2 1 7 449 733 27:35
🟨​ aws windows 7 1 7 417 743 23:15
❌​ aws-ucws linux 1 1 11 5 705 586 68:35
❌​ aws-ucws windows 1 1 6 11 5 662 597 65:20
🔄​ azure linux 3 1 9 449 732 26:15
🔄​ azure windows 2 2 9 415 742 23:18
💚​ azure-ucws linux 6 7 668 597 50:18
💚​ azure-ucws windows 6 7 631 608 51:57
💚​ gcp linux 2 9 440 738 25:10
💚​ gcp windows 2 9 406 748 23:17
30 interesting tests: 12 flaky, 7 KNOWN, 5 SKIP, 5 RECOVERED, 1 FAIL
Test Name aws linux aws windows aws-ucws linux aws-ucws windows azure linux azure windows azure-ucws linux azure-ucws windows gcp linux gcp windows
🟨​ TestAccept 🟨​K 🟨​K 🟨​K 🟨​K 🔄​f 💚​R 💚​R 💚​R 💚​R 💚​R
🔄​ TestAccept/bundle/config-remote-sync/job_pipeline_task ✅​p 🙈​s ✅​p 🙈​s 🔄​f 🙈​s ✅​p 🙈​s ✅​p 🙈​s
🔄​ TestAccept/bundle/config-remote-sync/job_pipeline_task/DATABRICKS_BUNDLE_ENGINE=direct ✅​p ✅​p 🔄​f ✅​p ✅​p
🙈​ TestAccept/bundle/deployment/bind/alert 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/generate/alert 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
💚​ TestAccept/bundle/invariant/no_drift 🙈​S 🙈​S 💚​R 💚​R 🙈​S 🙈​S 💚​R 💚​R 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/alerts/basic 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/alerts/with_file 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🔄​ TestAccept/bundle/resources/dashboards/change-serialized-dashboard/DATABRICKS_BUNDLE_ENGINE=direct ✅​p ✅​p ✅​p 🔄​f ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p
🔄​ TestAccept/bundle/resources/dashboards/change-serialized-dashboard/DATABRICKS_BUNDLE_ENGINE=terraform ✅​p ✅​p ✅​p 🔄​f ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p
🔄​ TestAccept/bundle/resources/dashboards/dataset-catalog-schema/DATABRICKS_BUNDLE_ENGINE=direct ✅​p ✅​p ✅​p 🔄​f ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p
🔄​ TestAccept/bundle/resources/dashboards/dataset-catalog-schema/DATABRICKS_BUNDLE_ENGINE=terraform ✅​p ✅​p ✅​p 🔄​f ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p
🙈​ TestAccept/bundle/resources/permissions 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions 🟨​K 🟨​K 💚​R 💚​R 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=direct 🟨​K 🟨​K 💚​R 💚​R
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=terraform 🟨​K 🟨​K 💚​R 💚​R
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions 🟨​K 🟨​K 💚​R 💚​R 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=direct 🟨​K 🟨​K 💚​R 💚​R
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=terraform 🟨​K 🟨​K 💚​R 💚​R
❌​ TestAccept/bundle/resources/postgres_endpoints/recreate 🙈​s 🙈​s ❌​F ❌​F 🙈​s 🙈​s 🙈​s 🙈​s 🙈​s 🙈​s
🔄​ TestAccept/bundle/resources/postgres_projects/recreate 🙈​s 🙈​s ✅​p 🔄​f 🙈​s 🙈​s 🙈​s 🙈​s 🙈​s 🙈​s
🔄​ TestAccept/bundle/resources/postgres_projects/recreate/DATABRICKS_BUNDLE_ENGINE=direct ✅​p 🔄​f
💚​ TestAccept/bundle/resources/synced_database_tables/basic 🙈​S 🙈​S 💚​R 💚​R 🙈​S 🙈​S 💚​R 💚​R 🙈​S 🙈​S
💚​ TestAccept/bundle/resources/synced_database_tables/basic/DATABRICKS_BUNDLE_ENGINE=direct 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/synced_database_tables/basic/DATABRICKS_BUNDLE_ENGINE=terraform 💚​R 💚​R 💚​R 💚​R
🔄​ TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=direct/DLT=yes/NBOOK=no/PY=no/READPLAN= ✅​p ✅​p ✅​p ✅​p ✅​p 🔄​f ✅​p ✅​p ✅​p ✅​p
🔄​ TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=direct/DLT=yes/NBOOK=no/PY=no/READPLAN=1 ✅​p ✅​p ✅​p ✅​p ✅​p 🔄​f ✅​p ✅​p ✅​p ✅​p
🔄​ TestAccept/bundle/templates/default-python/integration_classic 🔄​f ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p
🔄​ TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=direct/UV_PYTHON=3.13 🔄​f ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p
💚​ TestAccept/ssh/connection 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 50 slowest tests (at least 2 minutes):
duration env testname
8:01 azure-ucws windows TestAccept/bundle/invariant/migrate/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=database_instance.yml.tmpl
7:51 aws-ucws linux TestAccept/bundle/resources/synced_database_tables/basic/DATABRICKS_BUNDLE_ENGINE=terraform
7:35 aws-ucws windows TestAccept/bundle/invariant/migrate/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=database_catalog.yml.tmpl
7:31 aws-ucws linux TestAccept/bundle/invariant/migrate/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=database_instance.yml.tmpl
7:11 aws-ucws linux TestAccept/bundle/invariant/migrate/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=synced_database_table.yml.tmpl
6:50 aws-ucws linux TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=synced_database_table.yml.tmpl
6:49 aws-ucws windows TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=database_catalog.yml.tmpl
6:47 aws-ucws windows TestAccept/bundle/resources/synced_database_tables/basic/DATABRICKS_BUNDLE_ENGINE=direct
6:38 aws-ucws linux TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=database_catalog.yml.tmpl
6:23 aws-ucws windows TestAccept/bundle/resources/synced_database_tables/basic/DATABRICKS_BUNDLE_ENGINE=terraform
6:22 aws-ucws linux TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=database_instance.yml.tmpl
6:21 aws-ucws windows TestAccept/bundle/invariant/migrate/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=database_instance.yml.tmpl
6:10 azure-ucws windows TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=synced_database_table.yml.tmpl
6:07 aws-ucws windows TestAccept/bundle/invariant/migrate/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=synced_database_table.yml.tmpl
6:01 azure windows TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
5:56 aws-ucws windows TestAccept/bundle/resources/dashboards/delete-trashed-out-of-band/DATABRICKS_BUNDLE_ENGINE=direct
5:51 aws-ucws linux TestAccept/bundle/resources/synced_database_tables/basic/DATABRICKS_BUNDLE_ENGINE=direct
5:50 aws-ucws windows TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=database_instance.yml.tmpl
5:48 aws-ucws linux TestAccept/bundle/templates/default-python/combinations/serverless/DATABRICKS_BUNDLE_ENGINE=direct/DLT=no/NBOOK=no/PY=yes/READPLAN=1
5:47 gcp windows TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
5:35 gcp linux TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
5:30 azure-ucws windows TestAccept/bundle/invariant/migrate/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=synced_database_table.yml.tmpl
5:26 azure-ucws windows TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
5:23 gcp linux TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
5:18 aws-ucws linux TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
5:17 gcp windows TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
5:14 aws-ucws windows TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
5:08 aws-ucws windows TestAccept/bundle/resources/dashboards/delete-trashed-out-of-band/DATABRICKS_BUNDLE_ENGINE=terraform
5:03 aws-ucws linux TestAccept/bundle/templates/default-python/combinations/serverless/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=yes/NBOOK=yes/PY=yes/READPLAN=
4:47 aws-ucws windows TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=synced_database_table.yml.tmpl
4:35 azure-ucws windows TestAccept/bundle/resources/synced_database_tables/basic/DATABRICKS_BUNDLE_ENGINE=terraform
4:29 aws-ucws linux TestAccept/bundle/templates/default-python/combinations/serverless/DATABRICKS_BUNDLE_ENGINE=direct/DLT=no/NBOOK=no/PY=no/READPLAN=1
4:21 azure-ucws linux TestAccept/bundle/invariant/migrate/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=database_catalog.yml.tmpl
4:15 aws-ucws linux TestAccept/bundle/templates/default-python/combinations/serverless/DATABRICKS_BUNDLE_ENGINE=direct/DLT=no/NBOOK=no/PY=no/READPLAN=
4:08 azure-ucws linux TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=database_catalog.yml.tmpl
4:08 azure-ucws windows TestAccept/bundle/resources/synced_database_tables/basic/DATABRICKS_BUNDLE_ENGINE=direct
3:57 azure-ucws linux TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=database_instance.yml.tmpl
3:57 azure-ucws linux TestAccept/bundle/resources/synced_database_tables/basic/DATABRICKS_BUNDLE_ENGINE=terraform
3:46 azure-ucws linux TestAccept/bundle/invariant/migrate/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=synced_database_table.yml.tmpl
3:38 azure-ucws linux TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=synced_database_table.yml.tmpl
3:27 gcp windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:26 azure-ucws linux TestAccept/bundle/invariant/migrate/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=database_instance.yml.tmpl
3:25 azure-ucws linux TestAccept/bundle/resources/synced_database_tables/basic/DATABRICKS_BUNDLE_ENGINE=direct
3:23 aws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:19 azure-ucws windows TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=database_catalog.yml.tmpl
3:13 gcp windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
3:13 azure windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
3:12 azure-ucws windows TestAccept/bundle/invariant/migrate/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=database_catalog.yml.tmpl
3:08 azure-ucws windows TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=database_instance.yml.tmpl
3:08 aws-ucws windows TestAccept/bundle/templates/default-python/combinations/serverless/DATABRICKS_BUNDLE_ENGINE=direct/DLT=yes/NBOOK=no/PY=yes/READPLAN=

Copy link
Contributor

@denik denik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add config for invariant tests in acceptance/bundle/invariant?

- field: storage_root
reason: immutable

external_locations:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question - were you able to run 'make generate-direct' to update resources.generated.yml

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nope, it failed actually with

Traceback (most recent call last):
  File "/Users/andrew.nester/cli/bundle/direct/tools/generate_resources.py", line 12, in <module>
    import yaml
ModuleNotFoundError: No module named 'yaml'

which seems to be unrelated

"custom_tags":
"description": |-
PLACEHOLDER
github.com/databricks/databricks-sdk-go/service/catalog.AwsSqsQueue:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These descriptions should automatically be generated during the next CLI bump. Worth keeping an eye on because those descriptions will auto-generate our public documentation.

Copy link
Contributor

@shreyas-goenka shreyas-goenka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, lgtm except one concern. Can you please TAL whether we can get test coverage for this on some cloud as well. Otherwise this can be fragile.

}

func (e *ExternalLocation) GetURL() string {
// Return empty as external locations don't have a workspace URL
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

optional: Can we still return the revelant s3 URL? Or any UI equivalent we have for the resource. This provides a nice experience when running bundle summary.

Local = true
# External locations require actual storage credentials with cloud IAM setup
# which are environment-specific, so we only test locally with the mock server
Cloud = false
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we atleast run these tests on one cloud? Like just AWS maybe?

# which are environment-specific, so we only test locally with the mock server
Cloud = false
RecordRequests = false
RequiresUnityCatalog = true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

optional: You can also set RunsOnDbr = true no to run these tests on DBR. It's opt-in for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants