Skip to content

DDIR: give half_join an optional projection over the timed row - #1104

Merged
frankmcsherry merged 2 commits into
TimelyDataflow:master-nextfrom
frankmcsherry:ddir-half-join-row
Nov 21, 2026
Merged

frankmcsherry merged 2 commits into
TimelyDataflow:master-nextfrom
frankmcsherry:ddir-half-join-row

Conversation

@frankmcsherry

@frankmcsherry frankmcsherry commented Nov 21, 2026 •

Copy link
Copy Markdown
Member

Following #1103, a half join accepts an optional projection. The closure receives exactly the row that a bare half join produces, (key, ((left, right), time)). The time in that row is the coordinate-wise maximum of the lifted stream's data time and the arranged update's timestamp. The output update itself retains the stream update's timestamp.

For example, the two directions of a delta join can produce matching rows with:

let ab = a | lift | half_join(b, 0);
let ba = b | lift | half_join(a, 1, (k, ((y, x), t)) -> (k, ((x, y), t)));

Migration. Rewrite (k, l, r) -> (k2, v2) as (k, ((l, r), t)) -> (k2, (v2, t)) to preserve the previous output. A projection can also read the time, move it into the key, or drop it. The runtime applies the projection inside the half join and materializes only the requested data-time coordinates for its matches.

Most custom projections in an existing delta-join chain only need to rekey or reshape the values. They now have to bind and return t explicitly, which makes that common case wordier. A downstream half join checks the shape of its lifted input. A different timestamp of the right shape can still pass that check.

Verification. A two-coordinate test matches a stream update at (1, 0) with an arranged update at (0, 2). It observes the joined data time (1, 2) in a snapshot at (1, 0) and compares an explicit projection with a map on the bare result. Plan tests exercise bare, identity, and explicit projections through all five printer stages. Existing delta-join tests cover strictness, retractions, nested scopes, and multiway joins.

@frankmcsherry
frankmcsherry merged commit cc7c9b5 into TimelyDataflow:master-next Nov 21, 2026
7 checks passed
@frankmcsherry
frankmcsherry deleted the ddir-half-join-row branch November 21, 2026 01:23
frankmcsherry added a commit that referenced this pull request Nov 21, 2026
Since #1103 and #1104 a join's projection takes `(k, (l, r))` and a half join's takes `(k, ((l, r), t))`.
The recipes are rewritten to those forms, and their transcripts are unchanged.
The reference gives `join(other)` and `half_join(other, strict)` as the reference forms, with a projection as sugar for a `map`.
The `sum` recipe and the first program use `| sum`, and the reductions table lists it with its expansion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5
frankmcsherry added a commit that referenced this pull request Nov 21, 2026
Since #1103 and #1104 a join's projection takes `(k, (l, r))` and a half join's takes `(k, ((l, r), t))`.
The recipes are rewritten to those forms, and their transcripts are unchanged.
The reference gives `join(other)` and `half_join(other, strict)` as the reference forms, with a projection as sugar for a `map`.
The `sum` recipe and the first program use `| sum`, and the reductions table lists it with its expansion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5
frankmcsherry added a commit that referenced this pull request Nov 21, 2026
Since #1103 and #1104 a join's projection takes `(k, (l, r))` and a half join's takes `(k, ((l, r), t))`.
The recipes are rewritten to those forms, and their transcripts are unchanged.
The reference gives `join(other)` and `half_join(other, strict)` as the reference forms, with a projection as sugar for a `map`.
The `sum` recipe and the first program use `| sum`, and the reductions table lists it with its expansion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5
frankmcsherry added a commit that referenced this pull request Nov 21, 2026
Since #1103 and #1104 a join's projection takes `(k, (l, r))` and a half join's takes `(k, ((l, r), t))`.
The recipes are rewritten to those forms, and their transcripts are unchanged.
The reference gives `join(other)` and `half_join(other, strict)` as the reference forms, with a projection as sugar for a `map`.
The `sum` recipe and the first program use `| sum`, and the reductions table lists it with its expansion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5
frankmcsherry added a commit that referenced this pull request Nov 21, 2026
Since #1103 and #1104 a join's projection takes `(k, (l, r))` and a half join's takes `(k, ((l, r), t))`.
The recipes are rewritten to those forms, and their transcripts are unchanged.
The reference gives `join(other)` and `half_join(other, strict)` as the reference forms, with a projection as sugar for a `map`.
The `sum` recipe and the first program use `| sum`, and the reductions table lists it with its expansion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5
frankmcsherry added a commit that referenced this pull request Nov 21, 2026
* DDIR: recipes, one checked program per idiom

The user study found that the idioms that make DDIR programs fast or correct are hard to discover.
`examples/recipes` holds one small program for each, with comments that explain the idiom, a session that runs it on a few rows, and the session's recorded transcript.
`server/tests/recipes.rs` pipes each session into a server and checks its transcript at one and three workers.

The first eight:
- `sum`: an integer sum per key, via `join_function` into the diff and `count`.
- `top_k`: top k through a tree of `reduce(limit k)` groups.
- `retention`: a sliding window, retracting each row with `lift` and `delay`.
- `upsert`: a keyed table with put and delete, held in a root feedback variable.
- `left_join`: an outer join, with a declared enum for the missing side.
- `connected_components`: label propagation, prioritized with `delay`.
- `shortest_paths`: delta stepping, with buckets spaced in rounds.
- `triangles`: a delta join with `lift` and `half_join`.

The DDIR README links to the recipes' README.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5

* DDIR: a draft language reference

`LANGUAGE.md` gathers what was spread across the `pipe.rs` doc comment, corgi's op table and the README: collections and shapes, statements and sources, every collection operator, join fragments, lambdas and the scalar operations, time and scopes, iteration including root variables, and the language's design positions.
Each idiom links to the recipe that shows it.

The fragment doc comment in `pipe.rs` still said a one-variable fragment is keyed by `(x)`; since #1096 it is keyed by the variable itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5

* DDIR reference: operators by reference form and sugar

Each operator group names its reference form, and each sugar shows its expansion.
join_function is the reference for linear operators, join for joins, and reduce for reductions.
lift, lift_iter and half_join move to their own group, as they read times and are not functions of data alone.
Every expansion was run against its sugar on the server, with equal results.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5

* DDIR reference: count drops values, no value or lift_iter

This follows the ddir-prune branch, which removes value and lift_iter and has count project its values away.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5

* DDIR reference: inputs by label, bound at load; no import

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5

* DDIR recipes and reference: joins pass the joined row; use sum

Since #1103 and #1104 a join's projection takes `(k, (l, r))` and a half join's takes `(k, ((l, r), t))`.
The recipes are rewritten to those forms, and their transcripts are unchanged.
The reference gives `join(other)` and `half_join(other, strict)` as the reference forms, with a projection as sugar for a `map`.
The `sum` recipe and the first program use `| sum`, and the reductions table lists it with its expansion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant