Skip to content

[FLINK-40410][table-planner] Exclude VARIANT from constant folding - #28991

Open
manner wants to merge 1 commit into
apache:masterfrom
manner:FLINK-40410
Open

[FLINK-40410][table-planner] Exclude VARIANT from constant folding#28991
manner wants to merge 1 commit into
apache:masterfrom
manner:FLINK-40410

Conversation

@manner

@manner manner commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

What is the purpose of the change

This pull request excludes the VARIANT data type from constant folding VARIANT expressions. This previously resulted in a ClassCastException when running a query such as the following, where a constant expression followed a VARIANT expression in the projection.

SELECT PARSE_JSON('{}'), JSON_STRING(PARSE_JSON('{}'));

The ExpressionReducer.scala in flink-table-planner evaluates constant sub-expressions, so they get computed during compile-time and not during query run-time. It runs in two passes that must agree on which types get a slot in the intermediate (compile-time) result:

  • Phase 1 excludes unsupported "object literal" types (ROW/ARRAY/MAP/MULTISET/...) from compile-time evaluation. VARIANT was previously missing from this list and was therefore evaluated at compile-time and added to the intermediate result list.
  • Phase 2 has an specific case for VARIANT and treats VARIANT as excluded from compile-time evaluation. So a VARIANT constant gets evaluated and occupies a real slot in phase 1, but phase 2 doesn't know that and never advances its slot index. Then the next reduced expression reads the stale slot containing the VARIANT data and crashes on the bad cast.

Brief change log

  • Adds VARIANT to the exclude list of data types that use constant folding

Verifying this change

This change added tests and can be verified as follows:

  • Added test in JsonFunctionsITCase.java for VARIANT, where constant folding is enabled

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (yes / no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
  • The serializers: (yes / no / don't know)
  • The runtime per-record code paths (performance sensitive): (yes / no / don't know)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
  • The S3 file system connector: (yes / no / don't know)

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: Claude Sonnet 5

@flinkbot

flinkbot commented Aug 18, 2026

Copy link
Copy Markdown
Collaborator

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

Comment on lines +1041 to +1044
// Regression test for FLINK-40410: PARSE_JSON(f0) (VARIANT) precedes
// JSON_STRING(PARSE_JSON(f0)) (STRING) in a constant-folded projection. VARIANT
// is excluded from folding in ExpressionReducer, like ROW/ARRAY/MAP, and
// evaluated normally at runtime.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: this comment narrates the bug/ticket story rather than stating the timeless invariant. The test name already establishes it's a regression test, so the JIRA reference and blow-by-blow of what the old bug looked like can move to the commit message. Something tighter reads better long-term, e.g.:

Suggested change
// Regression test for FLINK-40410: PARSE_JSON(f0) (VARIANT) precedes
// JSON_STRING(PARSE_JSON(f0)) (STRING) in a constant-folded projection. VARIANT
// is excluded from folding in ExpressionReducer, like ROW/ARRAY/MAP, and
// evaluated normally at runtime.
// VARIANT must be excluded from constant folding, like ROW/ARRAY/MAP,
// otherwise a constant-folded projection reads a stale slot for it.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, I'd just drop the comment

@gustavodemorais gustavodemorais left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added one nit. Thanks for working and fixing this @manner

Comment on lines +1041 to +1044
// Regression test for FLINK-40410: PARSE_JSON(f0) (VARIANT) precedes
// JSON_STRING(PARSE_JSON(f0)) (STRING) in a constant-folded projection. VARIANT
// is excluded from folding in ExpressionReducer, like ROW/ARRAY/MAP, and
// evaluated normally at runtime.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, I'd just drop the comment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants