feat: DuckDB-only paired-histogram backend (DRC-3390 Stage B)

[danyelf]

Stage B — DuckDB-only paired-distribution backend + API contract

First runnable cut of the DRC-3390 paired-histograms work: the backend computes paired column distributions on DuckDB and returns a clean {status: "unsupported"} envelope for every other adapter. No UI yet — this PR is the backend plus the TypeScript API contract the rest of the stack builds on.

Stages: A = Storybook cells (#1398) · B = this PR · C = wire the UI to this contract · D = polyglot adapter coverage. Linear: DRC-3390.

What's in it

1. --inline-profile server flag

New flag (RECCE_INLINE_PROFILE), plumbed through RecceServerFlags.inline_profile. It gates the whole feature; off by default, so nothing changes for existing users.

2. Adapter capability layer + the profile task

Two new modules decide whether and how an adapter can compute approximate aggregates:

• capabilities.py — a one-row table: duckdb → full tier, every other adapter → disabled. The strategy router short-circuits non-DuckDB to the unsupported envelope instead of emitting SQL the engine can't parse.
• approx_aggregates.py — DuckDB renderers for approx_count_distinct / approx_quantile / approx_top_k; non-DuckDB raises UnsupportedAggregateError.

tasks/profile_distribution.py orchestrates the pipeline (HLL probe → percentile histograms → top-K), registered as RunType.PROFILE_DISTRIBUTION. Stage D adds the other adapters back to the table.

Per-env histogram edges (the key design call). Each env's histogram is built from its own quantiles — base_bin_edges + current_bin_edges, paired with base_density / current_density (density = Δfraction / span). The two edge arrays deliberately don't line up; Stage C overlays them on a shared axis. We only get approx percentiles back from the warehouse (not bin counts), so per-env edges are the only assumption-free rendering — and they're what lets base and current actually diverge. This snake_case wire shape maps to the camelCase baseBinEdges / currentBinEdges that #1398 consumes.

3. Run-type registry

profile_distribution is dispatchable but has no registry entry (no checklist title / icon / result view — its results feed schema-view cells). findByRunType now returns RegistryEntry | undefined, and the two dynamic callsites null-check. The registry stays the single source of truth for what's registered — no parallel run-type list.

Verifying it

Automated (runs in CI):

pytest tests/tasks/test_profile_distribution.py \
       tests/adapter/test_capabilities.py \
       tests/adapter/test_approx_aggregates.py
# 121 passing

Live — there's no UI yet, so drive the run API directly against a DuckDB project:

cd /path/to/duckdb-dbt-project
recce server      # API on :8000  (--inline-profile only gates the UI, not this path)

curl -s -X POST http://localhost:8000/api/runs \
  -H 'content-type: application/json' \
  -d '{"type":"profile_distribution","params":{"model":"orders"}}' | jq '.result'

Run against jaffle_shop_duckdb with a real base→current change (base = HEAD; current = a working-tree edit that drops returned orders and renames status), the per-env divergence comes through:

{
  "status": "ok", "strategy": "approx_all",
  "base_total": 999, "current_total": 903,          // returned orders dropped by current
  "columns": {
    "amount": {
      "kind": "histogram",
      "base_bin_edges":    [1, 7, 12.6, /* … */ 97],
      "current_bin_edges": [100, 703, 1276, /* … */ 9704],   // own quantiles per env — don't line up
      "base_density": [/* 11 */], "current_density": [/* 11, differs from base */]
    },
    "status": {
      "kind": "topk",                                // union of both envs' values
      "values": ["completed", "returned", /* … */ "ORDER-completed", "ORDER-placed", /* … */],
      "base_counts": null, "current_counts": null, "trimmed": false
    }
  }
}

Point the same project at a non-DuckDB target and the identical curl returns the envelope, not a crash:

{ "status": "unsupported", "reason": "Adapter 'snowflake' does not support APPROX_PERCENTILE.", "columns": {} }

Out of scope / follow-ups

• UI / schema-view integration → Stage C
• Polyglot SQL renderers + multi-row capability table → Stage D
• Per-column totals on the topK payload (Stage D, not needed here) — the discrete cell's counts mode wants base_total / current_total as a column-wide denominator for proportions. DuckDB's approx_top_k returns no counts, so Stage B always renders in ranks mode, which doesn't use totals. The field only becomes necessary once count-returning adapters land in Stage D. (Raised in the feat: paired histogram cells in Storybook (DRC-3390 Stage A) #1398 review; recorded here so it isn't lost.)
• Heavily-tied numeric columns render the tied point-mass as density 0 (no spike yet) — documented, follow-up later

[codecov]

Codecov Report

❌ Patch coverage is 87.67471% with 97 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
recce/tasks/profile_distribution.py	78.47%	96 Missing ⚠️
recce/cli.py	66.66%	1 Missing ⚠️

Files with missing lines	Coverage Δ
recce/adapter/approx_aggregates.py	100.00% <100.00%> (ø)
recce/adapter/capabilities.py	100.00% <100.00%> (ø)
recce/adapter/dbt_adapter/__init__.py	79.11% <ø> (+0.22%)	⬆️
recce/models/types.py	98.25% <100.00%> (+0.01%)	⬆️
recce/tasks/__init__.py	100.00% <100.00%> (ø)
tests/adapter/test_approx_aggregates.py	100.00% <100.00%> (ø)
tests/adapter/test_capabilities.py	100.00% <100.00%> (ø)
tests/tasks/test_profile_distribution.py	100.00% <100.00%> (ø)
recce/cli.py	68.88% <66.66%> (-0.01%)	⬇️
recce/tasks/profile_distribution.py	78.47% <78.47%> (ø)

... and 5 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[even-wei]

Code Review: PR #1399

SHA ec43d9e10 · Verdict NO-GO · Early review (Stages C & D to follow)

Blockers

None.

Issues

1. recce/tasks/profile_distribution.py:579 — the empty-columns "ok" path omits base_total/current_total, which the frozen TS contract declares as required.
   Evidence: line 579 returns {"status": "ok", "strategy": strategy, "columns": {}}; the main path (lines 680-684) returns those two fields, and ProfileDistributionOkResult in js/packages/ui/src/api/types/run.ts:261-262 declares base_total: number; current_total: number (non-optional). Reachable via a columns=[…] filter that matches nothing, or a model whose columns are all skipped types. Stage C reading result.base_total gets undefined. No test covers this branch (tests/tasks/test_profile_distribution.py).
   Pass C / F (+ E).

Notes

1. recce/tasks/profile_distribution.py:411 — datetime histogram edges are emitted as epoch seconds (the DRC-3504 epoch() cast), but ProfileDistributionHistogramPayload carries no marker separating numeric from datetime. Stage C must use out-of-band column type to format epoch values back to dates; worth recording in the frozen contract before C builds on it.
   Pass C.
2. recce/tasks/profile_distribution.py:676 — cap-degenerate columns emit base_counts: [] / current_counts: [], while the normal DuckDB categorical path emits null (line 661 → parse_topk_result returns None). Same adapter, two "no counts" encodings; Stage C must treat [] and null identically.
   Pass F.
3. recce/tasks/profile_distribution.py:551-561 — a model whose get_columns raises in both envs is swallowed to [] and returns status: "ok", columns: {} rather than surfacing an error. A nonexistent/broken model is indistinguishable from one with no profileable columns.
   Pass D.

---

What I verified as sound: the findByRunType overload + runTypeHasRef type-guard design (narrows run.type to RunTypeWithRef, a subset of registry keys, so RunResultPaneOss.tsx:147 / useRun.tsx:98 .RunResultView access stays non-undefined; the other dynamic callsites null-check). Histogram equal-area math, per-env edge divergence, tie→zero handling, DRC-3504 epoch cast, and the DRC-3507 rollback isolation are all covered by tests. The unsupported short-circuit, capability table, and SQL renderers (quantiles/k are constants, identifiers quoted via adapter.quote) carry no injection surface. All CI checks green on this SHA (Python 3.11–3.13, flake8, lint, type/build, CodeQL, codecov patch+project).

[danyelf]

Thanks for the early review @even-wei — addressed in 4009362e.

Stepping back, before I let the 🤖 say its bit -- looks like all the stuff was edge cases (columns with no items, empty columns). Indeed, I'm pretty sure that our main Histogram code crashes on datetime columns (or returns nothing); datetimes are annoying and need cleanup. Thanks for catching them!

--

Summary:

Issue 1 (:579, empty-columns "ok" path) — fixed. The early return now emits base_total: 0 / current_total: 0. The probe phase hasn't run at that point so the totals are genuinely unknown; 0 is the honest value and there are no columns to render against a denominator anyway. Added test_empty_columns_ok_payload_has_totals covering the columns=[…]-matches-nothing branch (asserts both totals present == 0).

Note 2 (:676, [] vs null counts) — fixed. Cap-degenerate columns now emit base_counts: null / current_counts: null, matching the normal categorical no-counts path. One "no counts" encoding per adapter; the empty values list is what flags degeneracy. Updated test_uuid_cap_degenerate_column accordingly.

Note 1 (:411, datetime epoch marker) — recorded in the contract. Added a doc block to ProfileDistributionHistogramPayload in run.ts spelling out that edges are always numeric but datetime columns carry epoch seconds (the DRC-3504 epoch() cast), with no in-payload marker — so Stage C must format edges using the column's out-of-band dbt type. No behavior change; this just freezes the assumption before C builds on it.

Note 3 (:551-561, get_columns raises → swallowed to []) — deferred to Stage D. Distinguishing a broken/nonexistent model from one with no profileable columns means deciding on an error envelope, which is Stage D's error-taxonomy scope. Left as-is intentionally; flagging here so it isn't lost.

All tests/tasks/test_profile_distribution.py pass (54); flake8 / black / isort / pnpm type:check clean.

[even-wei]

Code Review: PR #1399

SHA 6f6e48e5 · Verdict NO-GO

Verified locally: Python 122 passed; frontend pnpm type:check exit 0; full frontend suite 3850 passed / 5 skipped; flake8 + black --check clean. The registry overload refactor (findByRunType → RegistryEntry | undefined, runTypeHasRef as a RunTypeWithRef type guard) is sound — every dynamic call site is covered by either the type-guard narrowing or a null-check, and the registry test correctly switched from iterating RUN_TYPES to Object.keys(defaultRunTypeConfig). SQL-injection surface is closed (params resolve against the manifest / used as filters; identifiers quoted via adapter.quote(); aliases sanitized; quantiles/k range-validated).

Blockers

None.

Issues

1. recce/tasks/profile_distribution.py:448 — Base-first [:k] union truncation drops all current-only values when base already fills k slots, so high-cardinality categorical columns whose top-K diverges between base and current render current's new top values as entirely absent — defeating the base-vs-current comparison the cell exists to show.
   Evidence: union = list(seen.keys())[:k] with base inserted first (lines 441-448); each env's top-K is SQL-capped at k=12 (approx_top_k(col, 12)). If len(base_values) >= 12, union = all 12 base values and every current_ranks entry is None. This contradicts the docstring's "values present only in current's top-K appended on the right." No test catches it: every create_model(...) in the suite passes csv, csv (identical base==current), and _build_topk_ranks_payload is never unit-imported — the divergence path that the histogram builder got a dedicated regression test for (test_histogram_payload_base_and_current_diverge) has zero coverage for top-K.
   Pass A / E / F.

Notes

1. js/packages/ui/src/api/types/run.ts:191 (+ recce/tasks/profile_distribution.py:408) — The "frozen" histogram contract docs state edges "each hold 12 edges" / "NUM_BINS + 1 values," but _env_edges_and_density emits min + present-quantiles + max after sort, so tied/degenerate columns yield fewer (the type is correctly number[], and len(density) == len(edges) - 1 does hold). A Stage C consumer that indexes a fixed edges[11] on the doc's promise would be wrong. Pass C/F.
2. recce/tasks/profile_distribution.py:604-611 — execute() swallows all exceptions from both get_columns() calls into [], so a connection/permission/infra failure returns the same {status: "ok", columns: {}} as a model with no profilable columns. Off-by-default behind --inline-profile and feeds a non-critical schema cell, but it leans false-quiet for infra errors (get_columns itself raises a descriptive RecceException that is discarded here). Pass D.
3. recce/tasks/profile_distribution.py:489-547 — Stage-D-only latent contract drift: the counts-mode _build_topk_payload can set base_counts/current_counts to a whole-side None, but the TS ProfileDistributionTopKPayload types them as (number | null)[] and its doc claims they are "never wholly null"; it also unions current-first vs the ranks path's base-first. Dead in Stage B (DuckDB always → ranks); flagging so it isn't lost when count-returning adapters land in Stage D. Pass C/F.

[even-wei]

Claude Code Review: NO-GO. 1 issue (top-K base-vs-current divergence dropped on truncation; untested) + 3 notes. Full review in the PR comment above.