Added support for UC external locations (direct mode only) #4484
+1,253
−11
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| bundle: | ||
| name: test-bundle-$UNIQUE_NAME | ||
|
|
||
| resources: | ||
| external_locations: | ||
| test_location: | ||
| name: test_location_$UNIQUE_NAME | ||
| url: s3://test-bucket/path | ||
| credential_name: test_storage_credential | ||
| comment: "Test external location from DABs" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| bundle: | ||
| name: catalog-and-ext-loc-$UNIQUE_NAME | ||
|
|
||
| workspace: | ||
| root_path: ~/.bundle/$UNIQUE_NAME | ||
|
|
||
| resources: | ||
| catalogs: | ||
| test_catalog: | ||
| name: test_catalog_$UNIQUE_NAME | ||
| comment: "Test catalog for external locations" | ||
| properties: | ||
| owner: "dabs" | ||
| grants: | ||
| - principal: deco-test-user@databricks.com | ||
| privileges: | ||
| - USE_CATALOG | ||
| - CREATE_SCHEMA | ||
|
|
||
| external_locations: | ||
| test_location: | ||
| name: test_ext_location_$UNIQUE_NAME | ||
| url: s3://test-bucket/path | ||
| credential_name: test_storage_credential | ||
| comment: "Test external location from DABs" | ||
| skip_validation: true | ||
| read_only: false | ||
| grants: | ||
| - principal: deco-test-user@databricks.com | ||
| privileges: | ||
| - READ_FILES | ||
| - WRITE_FILES | ||
|
|
||
| targets: | ||
| development: | ||
| default: true | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,112 @@ | ||
|
|
||
| === Deploy bundle with catalog and external location | ||
| >>> [CLI] bundle plan | ||
| create catalogs.test_catalog | ||
| create catalogs.test_catalog.grants | ||
| create external_locations.test_location | ||
| create external_locations.test_location.grants | ||
|
|
||
| Plan: 4 to add, 0 to change, 0 to delete, 0 unchanged | ||
|
|
||
| >>> [CLI] bundle deploy | ||
| Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/[UNIQUE_NAME]/files... | ||
| Deploying resources... | ||
| Updating deployment state... | ||
| Deployment complete! | ||
|
|
||
| === Assert the catalog is created with grants | ||
| >>> [CLI] catalogs get test_catalog_[UNIQUE_NAME] | ||
| { | ||
| "name": "test_catalog_[UNIQUE_NAME]", | ||
| "comment": "Test catalog for external locations", | ||
| "properties": { | ||
| "owner": "dabs" | ||
| } | ||
| } | ||
|
|
||
| >>> [CLI] grants get catalog test_catalog_[UNIQUE_NAME] | ||
| { | ||
| "privilege_assignments": [ | ||
| { | ||
| "principal": "deco-test-user@databricks.com", | ||
| "privileges": [ | ||
| "CREATE_SCHEMA", | ||
| "USE_CATALOG" | ||
| ] | ||
| } | ||
| ] | ||
| } | ||
|
|
||
| === Assert the external location is created with grants | ||
| >>> [CLI] external-locations get test_ext_location_[UNIQUE_NAME] | ||
| { | ||
| "name": "test_ext_location_[UNIQUE_NAME]", | ||
| "url": "s3://test-bucket/path", | ||
| "credential_name": "test_storage_credential", | ||
| "comment": "Test external location from DABs" | ||
| } | ||
|
|
||
| >>> [CLI] grants get external_location test_ext_location_[UNIQUE_NAME] | ||
| { | ||
| "privilege_assignments": [ | ||
| { | ||
| "principal": "deco-test-user@databricks.com", | ||
| "privileges": [ | ||
| "READ_FILES", | ||
| "WRITE_FILES" | ||
| ] | ||
| } | ||
| ] | ||
| } | ||
|
|
||
| === Update external location comment | ||
| === Redeploy with updated comment | ||
| >>> [CLI] bundle plan | ||
| update external_locations.test_location | ||
|
|
||
| Plan: 0 to add, 1 to change, 0 to delete, 3 unchanged | ||
|
|
||
| >>> [CLI] bundle deploy | ||
| Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/[UNIQUE_NAME]/files... | ||
| Deploying resources... | ||
| Updating deployment state... | ||
| Deployment complete! | ||
|
|
||
| === Assert the external location comment is updated | ||
| >>> [CLI] external-locations get test_ext_location_[UNIQUE_NAME] | ||
| { | ||
| "name": "test_ext_location_[UNIQUE_NAME]", | ||
| "comment": "Updated external location from DABs" | ||
| } | ||
|
|
||
| === Update catalog comment | ||
| === Redeploy with updated catalog comment | ||
| >>> [CLI] bundle plan | ||
| update catalogs.test_catalog | ||
| update external_locations.test_location | ||
|
|
||
| Plan: 0 to add, 2 to change, 0 to delete, 2 unchanged | ||
|
|
||
| >>> [CLI] bundle deploy | ||
| Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/[UNIQUE_NAME]/files... | ||
| Deploying resources... | ||
| Updating deployment state... | ||
| Deployment complete! | ||
|
|
||
| === Assert the catalog comment is updated | ||
| >>> [CLI] catalogs get test_catalog_[UNIQUE_NAME] | ||
| { | ||
| "name": "test_catalog_[UNIQUE_NAME]", | ||
| "comment": "Updated catalog for external locations" | ||
| } | ||
|
|
||
| === Test cleanup | ||
| >>> [CLI] bundle destroy --auto-approve | ||
| The following resources will be deleted: | ||
| delete resources.catalogs.test_catalog | ||
| delete resources.external_locations.test_location | ||
|
|
||
| All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/[UNIQUE_NAME] | ||
|
|
||
| Deleting files... | ||
| Destroy complete! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| #!/bin/bash | ||
|
|
||
| envsubst < databricks.yml.tmpl > databricks.yml | ||
|
|
||
| CATALOG_NAME="test_catalog_${UNIQUE_NAME}" | ||
| EXT_LOCATION_NAME="test_ext_location_${UNIQUE_NAME}" | ||
|
|
||
| cleanup() { | ||
| title "Test cleanup" | ||
| trace $CLI bundle destroy --auto-approve | ||
| } | ||
| trap cleanup EXIT | ||
|
|
||
| title "Deploy bundle with catalog and external location" | ||
| trace $CLI bundle plan | ||
| trace $CLI bundle deploy | ||
|
|
||
| title "Assert the catalog is created with grants" | ||
| trace $CLI catalogs get "${CATALOG_NAME}" | jq "{name, comment, properties}" | ||
| trace $CLI grants get catalog "${CATALOG_NAME}" | jq --sort-keys | ||
|
|
||
| title "Assert the external location is created with grants" | ||
| trace $CLI external-locations get "${EXT_LOCATION_NAME}" | jq "{name, url, credential_name, comment}" | ||
| trace $CLI grants get external_location "${EXT_LOCATION_NAME}" | jq --sort-keys | ||
|
|
||
| title "Update external location comment" | ||
| update_file.py databricks.yml "Test external location from DABs" "Updated external location from DABs" | ||
|
|
||
| title "Redeploy with updated comment" | ||
| trace $CLI bundle plan | ||
| trace $CLI bundle deploy | ||
|
|
||
| title "Assert the external location comment is updated" | ||
| trace $CLI external-locations get "${EXT_LOCATION_NAME}" | jq "{name, comment}" | ||
|
|
||
| title "Update catalog comment" | ||
| update_file.py databricks.yml "Test catalog for external locations" "Updated catalog for external locations" | ||
|
|
||
| title "Redeploy with updated catalog comment" | ||
| trace $CLI bundle plan | ||
| trace $CLI bundle deploy | ||
|
|
||
| title "Assert the catalog comment is updated" | ||
| trace $CLI catalogs get "${CATALOG_NAME}" | jq "{name, comment}" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| Local = true | ||
| # External locations require actual storage credentials with cloud IAM setup | ||
| # which are environment-specific, so we only test locally with the mock server | ||
| Cloud = false | ||
|
Contributor
There was a problem hiding this comment. Should we atleast run these tests on one cloud? Like just AWS maybe? |
||
| RecordRequests = false | ||
| RequiresUnityCatalog = true | ||
|
Contributor
There was a problem hiding this comment. optional: You can also set RunsOnDbr = true no to run these tests on DBR. It's opt-in for now. |
||
|
|
||
| Ignore = [ | ||
| ".databricks", | ||
| "databricks.yml", | ||
| ] | ||
|
|
||
| [EnvMatrix] | ||
| DATABRICKS_BUNDLE_ENGINE = ["direct"] | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.