feat: Replace information schema with describe calls#1415
Open
tejassp-db wants to merge 16 commits intomainfrom
Open
feat: Replace information schema with describe calls#1415tejassp-db wants to merge 16 commits intomainfrom
tejassp-db wants to merge 16 commits intomainfrom
Conversation
Introduces support for Unity Catalog row filters across table, incremental, materialized view, and streaming table materializations to enable fine-grained row-level security [One Pager/PRD](https://drive.google.com/file/d/1gG1JeeEidcNGkuy9cScnbk0tWwwbLGps/view?usp=sharing) [Internal Doc] [Implementation Plan](https://drive.google.com/file/d/1dBxOzqgEWa6PnBKZb_AGR-TPfV0dvFAq/view?usp=sharing) [Internal Doc] ### Testing - Added functional tests to verify behaviour across the possible materialisation types [table/incremental/MV/ST] for CREATE/ALTER/DROP behaviour across V1/V2 materialisations - Added sufficient unit tests, to check correctness of helper function/macros ### Checklist - [x] Code has been tested in development - [x] Relevant tests have been added - [x] CHANGELOG.md has been updated with this feature
### Description Adds initial implementation of UC metric views as a materialization ### Checklist - [ ] I have run this code in development and it appears to resolve the stated issue - [x] This PR includes tests, or tests are not required/relevant for this PR - [x] I have updated the `CHANGELOG.md` and added information about my change to the "dbt-databricks next" section. --------- Co-authored-by: Eric Jang <eric.jang@databricks.com> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
## Description Users want to create Python UDFs using dbt's functions feature on Databricks, but the current dbt-databricks adapter restricts UDF creation to SQL only. This PR looks to add support for Python UDFs (of the [following form/format + more](https://learn.microsoft.com/en-us/azure/databricks/udf/unity-catalog)) ``` # def + explicit call -- for more complex logic def compute(value): return value * 2 return compute(value) # Inline return -- parameters are available as local variables return value * 2 ``` ## Checklist - [x] I have run this code in development and it appears to resolve the stated issue - [x] This PR includes tests, or tests are not required/relevant for this PR - [x] I have updated the `CHANGELOG.md` and added information about my change to the "dbt-databricks next" section.
Add alternate code path for metadata fetching in _describe_relation methods using DESCRIBE TABLE EXTENDED AS JSON (DBR 17.3+). Replaces 4 information_schema queries in IncrementalTableAPI and get_view_description in MaterializedViewAPI/ViewAPI. Falls back to legacy queries when capability is unavailable. PECOBLR-2546
Co-authored-by: Isaac
tejassp-db
changed the title
Coverage report
Click to see where and how coverage changed
This report was generated by python-coverage-comment-action |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Resolves #1270 ### Description Provides a solution to add support for configuring key-only `databricks_tags` for tables and columns. Utilizes existing databricks sql patterns, which treat values of key-only patterns as empty strings (when queried from `information_schema.tags` and also when applied via sql: `ALTER TABLE ... SET TAGS ('key_only_tag' = '', ...)`) ### Checklist - [x] I have run this code in development and it appears to resolve the stated issue - [x] This PR includes tests, or tests are not required/relevant for this PR - [x] I have updated the `CHANGELOG.md` and added information about my change to the "dbt-databricks next" section. --------- Signed-off-by: Can Bekleyici <can.bekleyici@deepl.com> Co-authored-by: Shubham Dhal <shubham.dhal@databricks.com>
Wrap long docstrings and expressions to satisfy E501, and add the missing return annotation on DatabricksRelation.is_foreign_table for mypy. Co-authored-by: Isaac
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Replace expensive
information_schemaqueries with a singleDESCRIBE TABLE EXTENDED ... AS JSONcall for metadata fetching in_describe_relationmethods. Gated behindDBRCapability.DESCRIBE_TABLE_EXTENDED_AS_JSON(DBR 17.3+), falls back to existing info_schema queries on older runtimes.Changes:
IncrementalTableAPI._describe_relation: replaces 4 info_schema queries (PK, FK, non-null constraints, column masks) with AS JSON parsingMaterializedViewAPI._describe_relation: replacesget_view_description(info_schema.views) with AS JSON parsingViewAPI._describe_relation: same as MVDatabricksDescribeJsonMetadataparser class with composite PK/FK supportis_describe_as_json_supported()gating method (checks HMS, foreign table, capability)is_foreign_tableproperty onDatabricksRelationdescribe_table_extended_as_jsonJinja macroTesting:
is_describe_as_json_supportedunit tests (UC, HMS, foreign table, no capability)Checklist
CHANGELOG.mdand added information about my change to the "dbt-databricks next" section.PECOBLR-2546