feat: support ASOF joins#23738
Conversation
github-actions
Bot
added
sql
# Conflicts: # datafusion/proto/src/physical_plan/mod.rs
|
Thank you for opening this pull request! Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch). Details |
xudong963
left a comment
xudong963
left a comment
There was a problem hiding this comment.
Thanks for the PR, the implementation is a complete vertical slice, looks promising.
Some todos in my mind:
- Add an ASOF section documenting syntax, left preservation, optional equality keys, operator directions, null behavior, bounded-only execution, single-partition behavior without equality keys, and nondeterministic selection among tied right rows.
- Add an
asof_join.sltcovering at least all four operators, grouped and ungrouped matching, nulls, USING, unmatched rows, invalid conditions, and EXPLAIN
github-actions
Bot
added
documentation
|
I think we should first agree on the high-level direction and then start polishing the implementation. We could ignore the implementation details and test coverage for now. I think the two most important questions are:
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #23738 +/- ##
==========================================
- Coverage 80.69% 80.59% -0.11%
==========================================
Files 1089 1091 +2
Lines 368447 370886 +2439
Branches 368447 370886 +2439
==========================================
+ Hits 297332 298920 +1588
- Misses 53405 54102 +697
- Partials 17710 17864 +154 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Thank you @2010YOUY01 joining the review. I agree we should settle the semantics and execution model first.
I propose Snowflake-style semantics: each left row selects the closest eligible right row within an optional equality-key group.
The initial operator uses an ordered merge: co-partition and order both inputs, then advance the right cursor monotonically while retaining the latest eligible candidate. This is O(L + R) after ordering with bounded join state. Other implementations could later share the same logical semantics. Does this direction make sense? |
alamb
left a comment
alamb
left a comment
There was a problem hiding this comment.
looks neat -- I skimmed it quickly
| | LogicalPlan::Aggregate(_) | ||
| | LogicalPlan::Sort(_) | ||
| | LogicalPlan::Join(_) | ||
| | LogicalPlan::AsOfJoin(_) |
There was a problem hiding this comment.
Why does AsOfJoin get its own logical plan? Isn't the choice of join algorithm typically done via a physical plan?
There was a problem hiding this comment.
I think ASOF joins and regular joins are different relational operations. If many planning or logical optimization tasks apply only to ASOF joins, splitting them would probably simplify the implementation; otherwise, they should remain combined.
(I’m not sure about the current state of the implementation, and which approach should we take.)
There was a problem hiding this comment.
Yes, the execution algorithm should remain a physical decision.
The separate logical node represents different relational semantics: for left ts = 10 and right ts = {1, 5}, a regular >= join returns both rows, while ASOF returns only 5. Replacing a regular Join’s physical algorithm with AsOfJoinExec would therefore change results.
The current implementation also needs ASOF-specific cardinality, type coercion, projection/filter pushdown, and must not participate in ordinary join reordering or input swapping. The logical node can still map to different physical implementations—ordered merge now, indexed probe later.
We could instead add an explicit match mode to Join, but then every generic join rule would need to account for it. I currently prefer a separate node for isolation, but I’m open to a generalized representation if that is preferred.
There was a problem hiding this comment.
Got it -- sorry -- I am still coming up to speed with AS OF (and how the relate to range joins).
Is there a standard syntax for AS OF joins? For example it seems like DuckDB has a different syntax https://duckdb.org/docs/current/guides/sql_features/asof_join
(no MATCH_CONDIITION) and we could maybe model it as a diferent join type 🤔
There was a problem hiding this comment.
I think @Xuanwo proposed to implement Snowflake syntax (with match_condition keyword)
https://docs.snowflake.com/en/sql-reference/constructs/asof-join
Personally I also think Snowflake syntax is better than the DuckDB syntax
-- DuckDB synatx
SELECT t.*, p.price
FROM trades t
ASOF JOIN prices p
ON t.symbol = p.symbol AND t.when >= p.when;
-- Snowflake syntax
SELECT t.*, p.price
FROM trades t
ASOF JOIN prices p
MATCH_CONDITION t.when >= p.when
ON t.symbol = p.symbol;
The reason is the in-equality predicate has different semantical meaning:
- equal condition: pre-filter all pairs before the ASOF join step
- in-equality condition: for all satisfied
pricestable rows, only return the closest match
Separating them is more intuitive given the special semantics of ASOF joins.
The downside is that we informally use DuckDB as our primary reference system, so following Snowflake here might depart from that convention.
|
Thank you @Xuanwo -- I think it is not likely I will be able to find time to carefully review a 4500 line PR It is really helpful to see the design running end to end Is there any chance you can break this one up into smaller PRs for easier review:
|
Yeah, agree. If we agree with the PR direction, then it's better to split into a couple of small PRs to make it easier to move forward. |
Thanks for explaining the rationale! I’m interested in helping further with this feature, now I need some time to think it through. |
|
For @alamb
Yes! That's exactly why I started a full end-to-end demo PR first. Proving that it works and using it as a starting point to split tasks makes much more sense nowadays.
Sure, will happy to do this. For @xudong963
Let's go. For @2010YOUY01
Thank you @2010YOUY01! I will start a series of stack PRs and get them merged one by one. This will allow you to have more time for thinking 😆 |
|
okay I got some idea how to implement it. Here are some thoughts: Semantics to implement
I think it's a good idea. DuckDB distinguishes between inner and outer ASOF joins, while Snowflake supports only one variant: left outer. I don't fully understand why Snowflake made this decision, but perhaps it is the most common pattern in practice. In any case, it would be preferable to start simple. Regarding ASOF join conditions, both DuckDB and Snowflake support a conjunction consisting of:
For example: SELECT t.*, p.price
FROM trades t
ASOF JOIN prices p
ON t.symbol = p.symbol AND t.when >= p.when; -- 1 ie cond + 1 eq condWe probably want to support the same join conditions. Algorithm to implement
Let's call this idea the repartition-based ASOF join. An alternative is the broadcast-based ASOF join, described below. I tend to think it's better to implement the broadcast-based algorithm in the first version, here are the reasons: (TLDR: assuming its common to have workloads with only inequality condition, but no equality ASOF join condition, the broadcast approach can fully utilize all CPUs, while the repartition based approach can't) I think both approaches will be useful in the long term, as they target different workload patterns.
For example, consider the join condition For workloads that can be perfectly repartitioned, the broadcast approach should not be much slower. Its main additional cost is maintaining and advancing more cursors over the broadcast side, which we could probably vectorize relatively easily. Implementation plan
|
|
FYI, my teammate @jonathanc-n is also working on the AsOf join implementation in our fork repo massive-com#65. We really want the feature to be done, which will speed up some materialization cases in our internal. He might have some thoughts about the topic and the future collaboration. |
5 participants
I understand there are many overlapping discussions and refactoring efforts regarding JOIN and ASOF JOIN. This PR aims to build the first workable foundation for all subsequent work. I am open to altering or rearranging the scope of this PR.
Parts of this PR were drafted with assistance from Codex (with
gpt-5.6-sol-max) and fully reviewed and edited by me. I take full responsibility for all changes.Stacked PRs
This PR remains the umbrella and end-to-end reference for ASOF JOIN. Its
implementation is now split into reviewable draft PRs refreshed to
origin/mainat47bd094c7.The child PRs supersede this PR as merge units and should be reviewed and
merged bottom-up.
Because GitHub cannot select a fork branch as the base of an upstream PR,
dependent PR pages show cumulative diffs against
main. Each PR body links itsisolated fork-to-fork diff.
xuanwo/asof-physicalAsOfJoinExec, execution state, properties, statistics, metrics, unit testsxuanwo/asof-logicalxuanwo/asof-sqlxuanwo/asof-dataframejoin_asof,join_asof_using, prelude exports, API testxuanwo/asof-protoxuanwo/asof-benchmarksdfbenchworkloads and physical Criterion benchmarkWhich issue does this PR close?
Rationale for this change
Time-series and ordered-event workloads often need to match each left row with the nearest eligible right row, optionally within equality-key groups. Expressing this through correlated subqueries is awkward and does not give the physical planner a dedicated ordered streaming execution contract.
This PR adds a first-class, left-preserving ASOF join for bounded inputs.
What changes are included in this PR?
ASOF JOIN ... MATCH_CONDITION (...)SQL planning and unparsing, plus logical-plan,LogicalPlanBuilder, andDataFrameAPIs.AsOfJoinExec, which consumes inputs ordered by equality keys and the match expression, preserves state across input/output batches, and supports<,<=,>, and>=nearest-match directions.Benchmark
Measured on feature commit
13ef0651144fda009e5bf5e4e268c753381949e5, on an Apple M4 Max (Darwin arm64), using four DataFusion partitions. The current head additionally mergesmain's SortMergeJoin protobuf serde migration; it does not change the ASOF execution path, benchmark harness, dataset, or build configuration, so the benchmark was not rerun.These are absolute results:
maincannot parse or plan the ASOF workload, so there is no equivalent main-branch baseline.Commands:
cargo run --release -p datafusion-benchmarks --bin dfbench -- asof --iterations 3 --partitions 4cargo bench -p datafusion-physical-plan --bench asof_join --features test_utils -- --quickThe physical-plan benchmark completed at approximately 33 ms for 100,000-row Int64, wide UTF-8, and dictionary payload cases.
Are there any user-facing changes?
Yes. Users can construct ASOF joins through SQL,
LogicalPlanBuilder, andDataFrame. The initial contract is left-preserving, supports optional equality keys plus one ordered match condition, and rejects unbounded inputs. Existing join behavior is unchanged.