Skip to content

DDIR: make join projections optional and pass the joined row - #1103

Merged
frankmcsherry merged 2 commits into
TimelyDataflow:master-nextfrom
frankmcsherry:ddir-join-row
Nov 20, 2026
Merged

frankmcsherry merged 2 commits into
TimelyDataflow:master-nextfrom
frankmcsherry:ddir-join-row

Conversation

@frankmcsherry

Copy link
Copy Markdown
Member

A DDIR equijoin can now be written without a projection. For matching inputs (k, l) and (k, r), left | join(right) produces (k, (l, r)). The optional projection receives that same row, so these expressions compute the same collection.

left | join(right, (k, (l, r)) -> (k, (l, r) add))
left | join(right) | map((k, (l, r)) -> (k, (l, r) add))

This changes existing projected joins. Migrate a binder such as (k, l, r) -> body to (k, (l, r)) -> body. The projection still runs inside the join operator. The examples, benchmark generators, and reviewed plan snapshots use the new binder.

A regression test checks the bare join, an explicit identity, and both forms of a rekeying projection on nested values across inserts and a retraction. The interactive and server test suites pass. Two regenerations reproduced every plan snapshot byte for byte, and all 33 Advent of Code programs passed planning.

@frankmcsherry
frankmcsherry merged commit 517d1be into TimelyDataflow:master-next Nov 20, 2026
7 checks passed
frankmcsherry added a commit that referenced this pull request Nov 21, 2026
Since #1103 and #1104 a join's projection takes `(k, (l, r))` and a half join's takes `(k, ((l, r), t))`.
The recipes are rewritten to those forms, and their transcripts are unchanged.
The reference gives `join(other)` and `half_join(other, strict)` as the reference forms, with a projection as sugar for a `map`.
The `sum` recipe and the first program use `| sum`, and the reductions table lists it with its expansion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5
frankmcsherry added a commit that referenced this pull request Nov 21, 2026
Since #1103 and #1104 a join's projection takes `(k, (l, r))` and a half join's takes `(k, ((l, r), t))`.
The recipes are rewritten to those forms, and their transcripts are unchanged.
The reference gives `join(other)` and `half_join(other, strict)` as the reference forms, with a projection as sugar for a `map`.
The `sum` recipe and the first program use `| sum`, and the reductions table lists it with its expansion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5
frankmcsherry added a commit that referenced this pull request Nov 21, 2026
Since #1103 and #1104 a join's projection takes `(k, (l, r))` and a half join's takes `(k, ((l, r), t))`.
The recipes are rewritten to those forms, and their transcripts are unchanged.
The reference gives `join(other)` and `half_join(other, strict)` as the reference forms, with a projection as sugar for a `map`.
The `sum` recipe and the first program use `| sum`, and the reductions table lists it with its expansion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5
frankmcsherry added a commit that referenced this pull request Nov 21, 2026
Since #1103 and #1104 a join's projection takes `(k, (l, r))` and a half join's takes `(k, ((l, r), t))`.
The recipes are rewritten to those forms, and their transcripts are unchanged.
The reference gives `join(other)` and `half_join(other, strict)` as the reference forms, with a projection as sugar for a `map`.
The `sum` recipe and the first program use `| sum`, and the reductions table lists it with its expansion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5
frankmcsherry added a commit that referenced this pull request Nov 21, 2026
Since #1103 and #1104 a join's projection takes `(k, (l, r))` and a half join's takes `(k, ((l, r), t))`.
The recipes are rewritten to those forms, and their transcripts are unchanged.
The reference gives `join(other)` and `half_join(other, strict)` as the reference forms, with a projection as sugar for a `map`.
The `sum` recipe and the first program use `| sum`, and the reductions table lists it with its expansion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5
frankmcsherry added a commit that referenced this pull request Nov 21, 2026
* DDIR: recipes, one checked program per idiom

The user study found that the idioms that make DDIR programs fast or correct are hard to discover.
`examples/recipes` holds one small program for each, with comments that explain the idiom, a session that runs it on a few rows, and the session's recorded transcript.
`server/tests/recipes.rs` pipes each session into a server and checks its transcript at one and three workers.

The first eight:
- `sum`: an integer sum per key, via `join_function` into the diff and `count`.
- `top_k`: top k through a tree of `reduce(limit k)` groups.
- `retention`: a sliding window, retracting each row with `lift` and `delay`.
- `upsert`: a keyed table with put and delete, held in a root feedback variable.
- `left_join`: an outer join, with a declared enum for the missing side.
- `connected_components`: label propagation, prioritized with `delay`.
- `shortest_paths`: delta stepping, with buckets spaced in rounds.
- `triangles`: a delta join with `lift` and `half_join`.

The DDIR README links to the recipes' README.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5

* DDIR: a draft language reference

`LANGUAGE.md` gathers what was spread across the `pipe.rs` doc comment, corgi's op table and the README: collections and shapes, statements and sources, every collection operator, join fragments, lambdas and the scalar operations, time and scopes, iteration including root variables, and the language's design positions.
Each idiom links to the recipe that shows it.

The fragment doc comment in `pipe.rs` still said a one-variable fragment is keyed by `(x)`; since #1096 it is keyed by the variable itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5

* DDIR reference: operators by reference form and sugar

Each operator group names its reference form, and each sugar shows its expansion.
join_function is the reference for linear operators, join for joins, and reduce for reductions.
lift, lift_iter and half_join move to their own group, as they read times and are not functions of data alone.
Every expansion was run against its sugar on the server, with equal results.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5

* DDIR reference: count drops values, no value or lift_iter

This follows the ddir-prune branch, which removes value and lift_iter and has count project its values away.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5

* DDIR reference: inputs by label, bound at load; no import

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5

* DDIR recipes and reference: joins pass the joined row; use sum

Since #1103 and #1104 a join's projection takes `(k, (l, r))` and a half join's takes `(k, ((l, r), t))`.
The recipes are rewritten to those forms, and their transcripts are unchanged.
The reference gives `join(other)` and `half_join(other, strict)` as the reference forms, with a projection as sugar for a `map`.
The `sum` recipe and the first program use `| sum`, and the reductions table lists it with its expansion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant