Repr types: Introduce repr types in all MIR code, and then change the MIR structs by ggevay · Pull Request #35084 · MaterializeInc/materialize

ggevay · 2026-02-19T12:15:50Z

This is pushing on the repr types work stream. It's on top of #35067, and stealing stuff from #35073.

The "Add soft_panic_or_log! canaries for repr type mismatches" commit turns some error logs into soft_panics, and the other commits attempt to make CI not run into any of these panics.

Nightlies:

https://buildkite.com/materialize/nightly/builds/15255 (stale)
https://buildkite.com/materialize/nightly/builds/15259 (stale)
https://buildkite.com/materialize/nightly/builds/15260 (stale)
https://buildkite.com/materialize/nightly/builds/15296 (stale)
https://buildkite.com/materialize/nightly/builds/15337 (stale)
https://buildkite.com/materialize/nightly/builds/15346 (stale)
https://buildkite.com/materialize/nightly/builds/15350 (stale)
https://buildkite.com/materialize/nightly/builds/15351 (stale)
https://buildkite.com/materialize/nightly/builds/15355 (limited to RQGs + SLTs)
https://buildkite.com/materialize/nightly/builds/15358 (stale)
https://buildkite.com/materialize/nightly/builds/15359 (slightly stale)
https://buildkite.com/materialize/nightly/builds/15366

Release qualifications:

https://buildkite.com/materialize/release-qualification/builds/1115 (slightly stale)

#35219 does some naming/cleanup that we'll keep separate (since it involves touching the *Func macros).

github-actions · 2026-02-19T12:15:59Z

Pre-merge checklist

The PR title is descriptive and will make sense in the git log.
This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post.
This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).

mgree

Okay---lots of suggestions/possibilities for improvement. Nothing seems wrong, so I'm marking it as approved.

Seems like there was some kind of instance/ghcr issue running nightly, but there may be benchmark issues as well.

I'll leave it up to you as to which of these changes are worth making---many are tiny refactors, but we'll want to rerun nightly and release qualification if we make them...

mgree · 2026-02-25T20:36:47Z

src/adapter/src/catalog.rs

                if let Ok(eval_result_datum) = mir.eval(&[], &arena) {
                    if let Some(return_styp) = return_styp {
-                        let mir_typ = mir.typ(&[]);
+                        let mir_typ = mir.sql_typ(&[]);


We could probably be more efficient here (we later compute mir_repr_scalar_type via round trip), but it's a test---I'm not worried about it.

mgree · 2026-02-25T20:47:59Z

src/adapter/src/coord.rs

 ) -> SqlRelationType {
    let mut typ = hir_expr.top_level_typ();
-    typ.backport_nullability_and_keys(&mir_expr.typ());
+    typ.backport_nullability_and_keys(&mir_expr.sql_typ());


We could alter this to work off mir_expr.typ() directly. Happy to save this for a later refactor.

Done, it was not hard

mgree · 2026-02-25T20:53:16Z

src/expr-parser/src/parser.rs

                syn::Lit::Str(l) => {
                    packer.push(Datum::from(l.value().as_str()));
-                    Ok(String::as_column_type())
+                    Ok(ReprColumnType::from(&String::as_column_type()))


Should as_column_type() have a SQL and repr version? Overkill?

Ideally, yes, but these 4 calls in this file would be the only usage, so I'm postponing this.

mgree · 2026-02-25T20:55:05Z

src/expr-test-util/src/lib.rs

-                    littyp.nullable(matches!(&litval[..], "null")),
+                Ok(Some(MirScalarExpr::literal_from_row(
+                    test_spec_to_row(std::iter::once((&litval[..], &littyp)))?,
+                    ReprScalarType::from(&littyp),


Confirming that we're not losing nullability information because literal_from_row can just look. (Caught me out!)

mgree · 2026-02-25T20:56:33Z

src/expr/src/relation/canonicalize.rs

                    }
                });
-                popped_expr.reduce(&column_types);
+                // Using Optimizer context is fine here: this is called from optimizer


I don't understand this comment.

Deleted.

(This comment was inserted by Claude, because I was telling it to be extra careful with what code is called from the optimizer vs. from upper levels when switching over stuff to repr types. So, it wanted to show me that it did check this carefully.)

mgree · 2026-02-25T21:41:02Z

src/transform/src/literal_constraints.rs

                        // Even better than what we were hoping for: Found contradicting
                        // literal constraints, so the whole relation is empty.
-                        relation.take_safely(Some(inp_typ));
+                        relation.take_safely(Some(inp_typ.clone()));


Unnecessary clone?

Yep, removed

mgree · 2026-02-25T21:44:34Z

src/transform/src/normalize_lets.rs

+                    // NB the ReprScalarType::eq ignorres nullability (correctly!)
+                    // since record field nullability can legitimately differ between the stored
+                    // type and the analysis-recomputed type.
                    if !new_type


All this could now just be new_type.column_types == typ.column_types.

Wait, ReprColumnType does not have a manual Eq impl, so this would not ignore nullability, while the current code ignores nullability.

And, indeed, if I change this to if new_type.column_types != typ.column_types, I get the following panic at startup:

internal error: failed to load bootstrap view: mz_object_history error: Optimizer(TransformError(Internal("scalar types do not match: [ReprColumnType { scalar_type: UInt64, nullable: false }, ReprColumnType { scalar_type: String, nullable: false }, ReprColumnType { scalar_type: String, nullable: false }, ReprColumnType { scalar_type: Jsonb, nullable: false }, ReprColumnType { scalar_type: String, nullable: true }, ReprColumnType { scalar_type: TimestampTz, nullable: false }, ReprColumnType { scalar_type: String, nullable: true }, ReprColumnType { scalar_type: UInt32, nullable: true }, ReprColumnType { scalar_type: String, nullable: true }, ReprColumnType { scalar_type: String, nullable: true }, ReprColumnType { scalar_type: String, nullable: true }, ReprColumnType { scalar_type: String, nullable: true }, ReprColumnType { scalar_type: String, nullable: true }, ReprColumnType { scalar_type: Array(MzAclItem), nullable: true }, ReprColumnType { scalar_type: String, nullable: false }, ReprColumnType { scalar_type: Bool, nullable: false }, ReprColumnType { scalar_type: String, nullable: true }, ReprColumnType { scalar_type: String, nullable: true }, ReprColumnType { scalar_type: UInt64, nullable: false }, ReprColumnType { scalar_type: String, nullable: false }, ReprColumnType { scalar_type: String, nullable: false }, ReprColumnType { scalar_type: Jsonb, nullable: false }, ReprColumnType { scalar_type: String, nullable: true }, ReprColumnType { scalar_type: TimestampTz, nullable: false }, ReprColumnType { scalar_type: String, nullable: false }, ReprColumnType { scalar_type: Bool, nullable: false }, ReprColumnType { scalar_type: String, nullable: true }] v [ReprColumnType { scalar_type: UInt64, nullable: false }, ReprColumnType { scalar_type: String, nullable: false }, ReprColumnType { scalar_type: String, nullable: false }, ReprColumnType { scalar_type: Jsonb, nullable: false }, ReprColumnType { scalar_type: String, nullable: true }, ReprColumnType { scalar_type: TimestampTz, nullable: false }, ReprColumnType { scalar_type: String, nullable: true }, ReprColumnType { scalar_type: UInt32, nullable: true }, ReprColumnType { scalar_type: String, nullable: true }, ReprColumnType { scalar_type: String, nullable: true }, ReprColumnType { scalar_type: String, nullable: true }, ReprColumnType { scalar_type: String, nullable: true }, ReprColumnType { scalar_type: String, nullable: true }, ReprColumnType { scalar_type: Array(MzAclItem), nullable: true }, ReprColumnType { scalar_type: String, nullable: false }, ReprColumnType { scalar_type: Bool, nullable: false }, ReprColumnType { scalar_type: String, nullable: false }, ReprColumnType { scalar_type: String, nullable: true }, ReprColumnType { scalar_type: UInt64, nullable: false }, ReprColumnType { scalar_type: String, nullable: false }, ReprColumnType { scalar_type: String, nullable: false }, ReprColumnType { scalar_type: Jsonb, nullable: false }, ReprColumnType { scalar_type: String, nullable: true }, ReprColumnType { scalar_type: TimestampTz, nullable: false }, ReprColumnType { scalar_type: String, nullable: false }, ReprColumnType { scalar_type: Bool, nullable: false }, ReprColumnType { scalar_type: String, nullable: false }]"))) Make sure that the schema name is specified in the builtin view's create sql statement. 5: __rustc::rust_begin_unwind at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/panicking.rs:697:5 6: core::panicking::panic_fmt at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/panicking.rs:75:14 7: mz_adapter::catalog::apply::<impl mz_adapter::catalog::state::CatalogState>::parse_builtin_views::{{closure}}::{{closure}} at ./src/adapter/src/catalog/apply.rs:1659:21 8: mz_adapter::catalog::apply::<impl mz_adapter::catalog::state::CatalogState>::parse_builtin_views::{{closure}} at ./src/adapter/src/catalog/apply.rs:1454:5

mgree · 2026-02-25T21:45:24Z

src/transform/src/reduce_elision.rs

                    .as_ref()
-                    .expect("Expression not well-typed");
+                    .expect("Expression not well-typed")
+                    .clone();


Unnecessary clone?

Indeed, removed

mgree · 2026-02-25T21:48:09Z

src/transform/src/union_cancel.rs

                    let relation_typ = base.typ();
-                    **base = MirRelationExpr::constant(vec![], relation_typ.clone());
-                    inputs[j] = MirRelationExpr::constant(vec![], relation_typ);
+                    **base = MirRelationExpr::Constant {


Switch these back to smart constructors?

mgree · 2026-02-25T21:48:18Z

src/transform/src/union_cancel.rs

                        let relation_typ = inputs[i].typ();
-                        inputs[i] = MirRelationExpr::constant(vec![], relation_typ.clone());
-                        inputs[j] = MirRelationExpr::constant(vec![], relation_typ);
+                        inputs[i] = MirRelationExpr::Constant {


Smart constructors?

…r types

…f trusting `Get`s, which now have canonicalized types

These fire when repr-type canonicalization does not fully prevent type mismatches, providing CI visibility without crashing production. - scalar.rs: base_eq_or_repr_eq_for_assertion panic -> soft_panic_or_log - context.rs: arrangement key fallback panic -> soft_panic_or_log - relation.rs: try_union trace -> soft_panic_or_log

Do the following easy switches from reduce(&[SqlColumnType]) to reduce_repr(&[ReprColumnType]): - simplify_to_literal / simplify_to_literal_with_result: It's ok to do the constant folding in MIR, because the result is untyped. - canonicalize.rs: 3 calls in canonicalize_equivalences, canonicalize_predicates, and replace_subexpr_and_reduce. These already had ReprColumnType available and were converting to SqlColumnType just to call reduce. The conversion is now removed entirely. - plan_index_exprs in query.rs: Index key expressions should have repr types so optimizer code can find matching indexes. - WebhookValidation::reduce_expression in plan.rs: Webhook validation expressions are used at runtime where repr types are the native currency.

Several transform files were storing SqlRelationType obtained via typ() (which converts stored ReprRelationType to SqlRelationType), only to convert back to ReprRelationType later. Switch these to use repr_typ() and store ReprRelationType directly, eliminating unnecessary round-trips. Files changed: projection_lifting.rs, projection_pushdown.rs, cse/anf.rs, union_cancel.rs, demand.rs, dataflow.rs.

…ype analysis

Switch Scope struct in expr-test-util to use ReprRelationType instead of SqlRelationType. Update all insert/set/get/iter signatures accordingly. build_get no longer needs conversion, build_let uses repr_typ(), and reverse_syntax_override stores ReprRelationType directly. Update doc comments in fusion/filter.rs and predicate_pushdown.rs to construct MirRelationExpr::Constant directly with ReprRelationType instead of using MirRelationExpr::constant() with SqlRelationType.

…of col types

ggevay · 2026-02-26T14:02:47Z

Addressed all the comments.

Waiting on a new Nightly.

Discussing the Release Qualification failures here.

ggevay added the A-optimization Area: query optimization and transformation label Feb 19, 2026

ggevay changed the title ~~DNM Burnin down errors from base_eq_or_repr_eq_for_assertion~~ DNM Burning down errors from base_eq_or_repr_eq_for_assertion Feb 19, 2026

ggevay mentioned this pull request Feb 19, 2026

sql: MIR transform to canonicalize SQL types by round tripping through repr #35067

Open

ggevay force-pushed the repr-types-canonicalize-xform--gabor-debugging branch 9 times, most recently from a9ab2a6 to 71638ef Compare February 25, 2026 10:01

ggevay changed the title ~~DNM Burning down errors from base_eq_or_repr_eq_for_assertion~~ DNM Burning down repr type errors Feb 25, 2026

ggevay changed the title ~~DNM Burning down repr type errors~~ [DNM] Burning down repr type errors Feb 25, 2026

ggevay force-pushed the repr-types-canonicalize-xform--gabor-debugging branch 2 times, most recently from 8e0ab4e to 42b4805 Compare February 25, 2026 12:46

ggevay changed the title ~~[DNM] Burning down repr type errors~~ Repr types: Introduce repr types in all MIR code, and then change the MIR structs Feb 25, 2026

ggevay force-pushed the repr-types-canonicalize-xform--gabor-debugging branch from 42b4805 to 7575ddb Compare February 25, 2026 14:02

mgree mentioned this pull request Feb 25, 2026

sql: post repr types: Rename output_type() to output_sql_type() on all scalar funcs #35219

Open

mgree marked this pull request as ready for review February 25, 2026 20:17

mgree requested review from a team as code owners February 25, 2026 20:17

mgree requested a review from SangJunBak February 25, 2026 20:17

mgree approved these changes Feb 25, 2026

View reviewed changes

mgree and others added 5 commits February 26, 2026 10:19

MIR transform to canonicalize SQL types by round tripping through rep…

1962dca

…r types

make arrangement resolution try to fall back; rewrite tests

99b2986

have persist_fast_path_order actually look up catalog types instead o…

74fbefd

…f trusting `Get`s, which now have canonicalized types

expect that Get's ID is also imported

834b8c4

ggevay and others added 22 commits February 26, 2026 10:19

Change MIR structs to repr types

1aa7082

Fix mzreflect glitch

ec9cd31

Eliminate SQL types from FoldConstants

f4c7749

Eliminate the repr type fallback in CollectionBundle::arrangement

a96cce5

drop reprize, push repr types through index imports and exports

145697d

expr-parser: more repr types

090f888

custom eq and ord instances, drop Repr*::base_eq

c75d353

canonicalize typ() methods; drop repr_canonicalize; drop SqlRelationT…

e4413ef

…ype analysis

Use smart constructors for MirScalarExpr::Literal

61658b3

change MRE::constant to take a repr type

4feb562

fix lints

779b8e9

renames and cleanup

e1cc323

turn mz_repr::... into imports and unqualified uses

bc61c08

Switch HIR lowering and anti_lookup/lookup to repr types

3807120

cargo fmt

b37e4b4

Address Michael's comments

11caa1f

Switch backport_nullability to repr types

0d6a811

Address minor comments

f02f953

AggregateExpr::on_unique -- more repr types and scalar types instead …

02d5a80

…of col types

ggevay force-pushed the repr-types-canonicalize-xform--gabor-debugging branch 2 times, most recently from 737d089 to d1e1bee Compare February 26, 2026 13:33

ggevay force-pushed the repr-types-canonicalize-xform--gabor-debugging branch 2 times, most recently from 579e0ca to ba812f5 Compare February 26, 2026 14:10

Address more minor comments

4931c50

ggevay force-pushed the repr-types-canonicalize-xform--gabor-debugging branch from ba812f5 to 4931c50 Compare February 26, 2026 14:31

ggevay merged commit 52af3ba into MaterializeInc:main Feb 26, 2026
136 checks passed

Conversation

ggevay commented Feb 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Feb 19, 2026

Pre-merge checklist

Uh oh!

mgree left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ggevay Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ggevay commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ggevay commented Feb 19, 2026 •

edited

Loading

ggevay Feb 26, 2026 •

edited

Loading

ggevay commented Feb 26, 2026 •

edited

Loading