[SPARK-59628][SQL] Report a missing MATCH_CONDITION in ASOF JOIN instead of parsing ASOF as an alias - #58966
Open
AMC-hawk wants to merge 1 commit into
Open
[SPARK-59628][SQL] Report a missing MATCH_CONDITION in ASOF JOIN instead of parsing ASOF as an alias#58966AMC-hawk wants to merge 1 commit into
AMC-hawk wants to merge 1 commit into
Conversation
…ead of parsing ASOF as an alias MATCH_CONDITION is required for an ASOF join, but ASOF is a non-reserved keyword. When MATCH_CONDITION was missing and the left relation had no alias, the parser read ASOF as that alias, so `SELECT * FROM t ASOF JOIN u ON t.a = u.a` silently ran as a plain inner join of `t AS asof` with `u` and returned wrong results. With an alias on the left relation the same mistake failed with a generic PARSE_SYNTAX_ERROR. A table alias no longer takes ASOF implicitly when JOIN follows it, and MATCH_CONDITION is optional in the grammar so that AstBuilder can report a missing one by name with the new ASOF_JOIN_MATCH_CONDITION_MISSING error. An alias named asof is still allowed when written as `t AS asof`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
MATCH_CONDITIONis required for an ASOF join, butASOFis a non-reserved keyword, so it is also a legal table alias. WhenMATCH_CONDITIONis missing, the ASOF branch ofjoinRelationcannot match, and the parser falls back to readingASOFas the alias of the left relation:This PR makes that case fail with a clear error, in two parts:
tableAliasno longer takesASOFas an implicit alias when the next token isJOIN. This uses a semantic predicate,isAsofJoinAhead(), in the same way the grammar already usesisOperatorPipeStart(). An explicitASis unaffected, sot AS asof JOIN ustill aliases the relation, andasofremains usable as an identifier everywhere else.MATCH_CONDITIONis made optional inasofJoinCriteria, so that omitting it reachesAstBuilder.withAsOfJoin, which raises the newASOF_JOIN_MATCH_CONDITION_MISSINGerror (SQLSTATE42601). It is still required: omitting it always fails. The check runs after the existing feature flag check, so with ASOF disabled users still seeUNSUPPORTED_FEATURE.ASOF_JOINfirst.Part 1 alone would turn the silent inner join into a generic
PARSE_SYNTAX_ERROR; part 2 makes the error name the missing clause, as the JIRA asks.Making
ASOFa reserved keyword would also fix the ambiguity, but would break every existing query that usesasofas an identifier. The predicate only refuses the one ambiguous shape, an implicit alias immediately beforeJOIN.Why are the changes needed?
An ASOF join returns the single nearest right row for each left row. A plain inner join returns every matching row. Parsing a malformed ASOF join as an inner join therefore returns silently wrong results, often many more rows on time series data, with no error or warning.
The behavior was also inconsistent. Only the default join type with an unaliased left relation was misread; with an alias or with
LEFT, the same mistake failed with a genericPARSE_SYNTAX_ERROR ... at or near 'ON'that did not mentionMATCH_CONDITION.This is one of the two parser stage defects tracked by SPARK-59626.
Does this PR introduce any user-facing change?
Yes. An ASOF join without
MATCH_CONDITIONnow fails with a clear error in every form, instead of running as an inner join or failing with a generic syntax error. The ASOF JOIN syntax is disabled by default (spark.sql.join.asofJoin.enabled) and is unreleased.With ASOF enabled:
SELECT * FROM t ASOF JOIN u ON t.a = u.a: before, ran ast AS asofINNER JOINu. After,ASOF_JOIN_MATCH_CONDITION_MISSING.SELECT * FROM t ASOF JOIN u: before, ran ast AS asofINNER JOINu. After,ASOF_JOIN_MATCH_CONDITION_MISSING.SELECT * FROM t x ASOF JOIN u ON t.a = u.a: before,PARSE_SYNTAX_ERRORat'ON'. After,ASOF_JOIN_MATCH_CONDITION_MISSING.SELECT * FROM t LEFT ASOF JOIN u ON t.a = u.a: before,PARSE_SYNTAX_ERRORat'ON'. After,ASOF_JOIN_MATCH_CONDITION_MISSING.Unchanged: valid ASOF joins;
t AS asof JOIN u ...andt `asof` JOIN u ..., which still alias the relation asasof;FROM t asof; andasofas a column name. A query that usedasofas an implicit alias directly beforeJOINmust now writeAS asofor quote it.How was this patch tested?
New tests:
PlanParserSuite:asof join - missing match condition rejectedcovers five shapes (unaliased withON, aliased withON,USING,LEFTwith no condition, no condition at all) and checks the error condition, SQLSTATE and query context.asof join - an alias named asof is still allowed when written with ASchecks thatt as asof join u ...andt asofstill parse to an aliased relation.join-asof-grammar.sql: three new golden queries for the unaliased case, the case with no join condition, andtrades AS asof JOIN quotes ...returning rows. The existing aliased case now reportsASOF_JOIN_MATCH_CONDITION_MISSINGinstead ofPARSE_SYNTAX_ERROR; this is the only change to existing expected output. Golden files were regenerated withSPARK_GENERATE_GOLDEN_FILES=1.Existing tests, since
tableAliasis used by almost every query:Was this patch authored or co-authored using generative AI tooling?
Generated-by: Yes, Claude