1278 recursive databricks and spark checks and custom queries by rob-h-w · Pull Request #1311 · datacontract/datacontract-cli

rob-h-w · 2026-06-16T16:17:04Z

Supports recursive databricks & spark checks and custom queries, replaces the ibis implementation that requires write permissions on connection until a new release of ibis stops doing that.

Tests pass (uv run pytest)
Code formatted (uv run ruff check --fix && uv run ruff format)
[-] README.md updated (if relevant)
CHANGELOG.md entry added

I don't see a README.md update that's in scope.

jochenchrist · 2026-08-01T16:23:43Z

@rob-h-w I see this PR is still in draft. Give me a ping whenever you feel this is ready to review.

To the Contribution>Troubleshooting section. Because I ran into that issue after checking out.

This is the beginning of an attempt to support ODCS' nested constraint and quality definition capabilities in `test` subcommands, where the underlying technology supports it. See [the relevant issue](datacontract#1278). Add recursive traversal of nested struct and array-of-struct fields when generating ibis quality checks, scoped to verified backends only. Check generation (create_checks.py): - Add _iter_property_paths() to recursively yield (model, field_path, prop, is_nested) tuples for nested struct fields and array item models - Struct recursion enabled for dataframe and databricks; array recursion for dataframe only - Nested SQL quality checks emit MetricType.UNSUPPORTED with a warning preset on all other backends - Use get_server_type() instead of server.type so imported DCS contracts with type="custom" are resolved correctly Check execution (ibis_check_execute.py): - Add _resolve_expr() / _resolve_nested_expr() for dotted-path ibis expressions - Add _resolve_dtype() / _field_present() for nested schema introspection - Update _run_present() to reuse the already-resolved model schema rather than re-fetching the table, fixing a case-sensitivity failure on Oracle - Update _run_type(), _run_freshness(), _run_duplicate(), _missing_expr(), _valid_expr(), _invalid_expr(), _samples_for() to accept resolved expressions instead of bare column names Spark temp view materialisation (kafka.py, connect.py): - Add add_spark_nested_views() to create {model}__{field} Spark temp views for nested struct fields and exploded array-of-struct items - Call add_spark_nested_views_for_contract() in the dataframe and Databricks-via-Spark connection paths before creating the ibis pyspark backend Tests: - tests/fixtures/dataframe/datacontract_nested.yaml: nested struct + array fixture - tests/test_create_checks_nested.py: unit tests for recursive generation and backend gating - tests/test_ibis_check_execute.py: regression tests for Oracle-style presence check without extra table lookup - tests/test_test_dataframe.py: Spark integration pass/fail for nested struct, nested SQL quality, and array-item checks - tests/test_test_databricks.py: unit test confirming nested struct SQL enabled and array recursion suppressed for Databricks Created with Claude Sonnet 4.6.

…atacontract#1278) Implement support for recursive nested struct and array checks on Databricks, with zero-permission requirements (SELECT-only). This enables data contract validation on read-only SQL warehouses without requiring CREATE VOLUME or CREATE TABLE permissions. Changes: - New module `databricks_nested_models.py`: CTE-based virtual model generation for array item checks. Uses `LATERAL VIEW OUTER explode_outer()` to expose nested array elements as queryable tables without creating real volumes. - Modified `_connect_databricks()` in `connect.py`: Introduced `_NoVolumeBackend` subclass that overrides `_post_connect()` with a no-op, bypassing ibis' default `CREATE VOLUME IF NOT EXISTS` call. Connection succeeds on read-only warehouses. - Updated `connect_ibis()` Databricks branch: Builds and attaches virtual model CTE queries to the backend connection for downstream table resolution. - Enabled array recursion for Databricks: Added "databricks" to `_SUPPORTED_NESTED_ARRAY_SERVER_TYPES` in `create_checks.py`, matching feature parity with Dataframe backend. - Enhanced `_resolve_table()` in `ibis_check_execute.py`: Falls back to virtual model CTE queries before attempting list_tables(), allowing nested array models (e.g., `orders__items`) to resolve via pre-built WITH clauses. - Test updates: Rewrote Databricks auth tests to patch the correct backend method, added `test_no_create_volume_on_connect` to verify volume creation is skipped, flipped nested array expectations to enable checks on array items. - New test file `test_connect_databricks_virtual_models.py`: Unit tests for CTE query generation and schema filtering logic. Result: 52 real-world data contract checks now pass against Databricks without any CREATE/WRITE operations. Recursive struct checks (dotted paths) and recursive array item checks (CTE virtual models) both fully supported.

Per the PR template.

rob-h-w · 2026-09-02T05:59:45Z

@rob-h-w I see this PR is still in draft. Give me a ping whenever you feel this is ready to review.

Hi, @jochenchrist, I think this is worth a look now. Apologies for the delay.

The comments in _connect_databricks and test_connect_databricks referred to a "_NoVolumeBackend subclass" that overrides _post_connect via MRO, but the implementation temporarily monkey-patches Backend._post_connect with a no-op lambda inside _databricks_connect and restores it in a finally block. No subclass or MRO override is involved. Update the docstring and test comments to describe what the code actually does. Co-authored-by: Copilot using Claude Opus 4.7

jochenchrist · 2026-09-03T08:20:39Z

The root cause is fixed with ibis-project/ibis#11962 (yet not released), correct?

jochenchrist · 2026-09-03T08:24:11Z

            database_name = ".".join(filter(None, [server.catalog, server.schema_]))
            if database_name:
                spark.sql(f"USE {database_name}")
+            from datacontract.engines.ibis.connections.kafka import add_spark_nested_views_for_contract


We should not have a kafka package dependency in the non-kafka path.

mahic · 2026-09-04T12:14:16Z

@jochenchrist Ibis has an upcoming release ibis-project/ibis#12102 which probably would include the ibis-project/ibis#11962 fix

jschoedl · 2026-09-10T14:53:16Z

+#### Linux system package for postgres/psycopg-based tests
+
+**Ubuntu/Debian:**
+```bash
+sudo apt-get update
+sudo apt-get install -y libpq-dev
+```
+**Fedora/RHEL:**
+```bash
+# Fedora/RHEL:
+sudo dnf install -y postgresql-devel
+# Arch:
+sudo pacman -S postgresql-libs
+```
+


This should no longer be needed on current main, afaik.

- Replace exploded temp views and CTE virtual models with array predicates - Collapse item_model dead variable back to model - Remove unreachable array_path.rstrip('.') - Fix case-folding on value paths (_struct_path, _resolve_nested_expr) - Route nested unique checks through the same predicate as samples collection - Handle unsupported cases gracefully: array-item quality rules warn, nested physicalType checks skip - Restore test_connect_databricks.py to main's version (unrelated rewrite) - Add tests for case-folding across array hops and nested unique sampling

jschoedl · 2026-09-10T15:12:02Z

Hi @rob-h-w, thank you for your PR!

As far as I can tell, this PR is not really blocked by the Ibis release since the workaround for the Ibis bug is already at main anyways and doesn't get introduced here.

I took a look at the code and took the liberty to change a few parts of the approach, although the main part stayed. Most importantly, you had added myModel__items models which were then made queryable using a Spark temp view or SQL strings attached to a Databricks connection. Now, model names only the real table, and we instead compile a predicate like SIZE(FILTER(items, i -> i.sku IS NULL)) > 0 over the elements.

This seems a bit more robust to me than messing with the Databricks connection. It also fixed a few things I ran into while testing:

an order with an empty items array failed required: true on an item field, because explode_outer produces a NULL row for it
--filter was silently ignored for the nested checks, since orders__items is never a key in the filter map
counts came back in item rows while every other check counts table rows, and both get summed into one quality score

jschoedl added the waiting-for-response label Aug 14, 2026

rob-h-w force-pushed the 1278-recursive-databricks-and-spark-checks branch from 8df1ac9 to 4dcdecf Compare August 24, 2026 16:23

jschoedl removed the waiting-for-response label Aug 26, 2026

rob-h-w added 4 commits September 1, 2026 13:24

Add a Linux Dev Dependency

c879313

To the Contribution>Troubleshooting section. Because I ran into that issue after checking out.

Update Formatting

669acc7

Per the PR template.

rob-h-w force-pushed the 1278-recursive-databricks-and-spark-checks branch from 4dcdecf to 669acc7 Compare September 1, 2026 11:41

rob-h-w marked this pull request as ready for review September 2, 2026 06:38

jochenchrist reviewed Sep 3, 2026

View reviewed changes

Merge branch 'main' into 1278-recursive-databricks-and-spark-checks

ee59f96

jschoedl reviewed Sep 10, 2026

View reviewed changes

Navigation Menu

Uh oh!

1278 recursive databricks and spark checks and custom queries - #1311

1278 recursive databricks and spark checks and custom queries#1311
rob-h-w wants to merge 7 commits into
datacontract:mainfrom
rob-h-w:1278-recursive-databricks-and-spark-checks

rob-h-w commented Jun 16, 2026 •

edited

Loading

Uh oh!

jochenchrist commented Aug 1, 2026

Uh oh!

rob-h-w commented Sep 2, 2026

Uh oh!

jochenchrist commented Sep 3, 2026

Uh oh!

jochenchrist Sep 3, 2026

Uh oh!

mahic commented Sep 4, 2026

Uh oh!

jschoedl Sep 10, 2026 •

edited

Loading

Uh oh!

jschoedl commented Sep 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

rob-h-w commented Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jochenchrist commented Aug 1, 2026

Uh oh!

rob-h-w commented Sep 2, 2026

Uh oh!

jochenchrist commented Sep 3, 2026

Uh oh!

jochenchrist Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

mahic commented Sep 4, 2026

Uh oh!

jschoedl Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jschoedl commented Sep 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

rob-h-w commented Jun 16, 2026 •

edited

Loading

jschoedl Sep 10, 2026 •

edited

Loading