Repository navigation
DDIR: make join projections optional and pass the joined row - #1103
Merged
frankmcsherry merged 2 commits intoNov 20, 2026
Merged
Conversation
This was referenced Nov 21, 2026
frankmcsherry
added a commit
that referenced
this pull request
Nov 21, 2026
Since #1103 and #1104 a join's projection takes `(k, (l, r))` and a half join's takes `(k, ((l, r), t))`. The recipes are rewritten to those forms, and their transcripts are unchanged. The reference gives `join(other)` and `half_join(other, strict)` as the reference forms, with a projection as sugar for a `map`. The `sum` recipe and the first program use `| sum`, and the reductions table lists it with its expansion. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5
frankmcsherry
added a commit
that referenced
this pull request
Nov 21, 2026
Since #1103 and #1104 a join's projection takes `(k, (l, r))` and a half join's takes `(k, ((l, r), t))`. The recipes are rewritten to those forms, and their transcripts are unchanged. The reference gives `join(other)` and `half_join(other, strict)` as the reference forms, with a projection as sugar for a `map`. The `sum` recipe and the first program use `| sum`, and the reductions table lists it with its expansion. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5
frankmcsherry
added a commit
that referenced
this pull request
Nov 21, 2026
Since #1103 and #1104 a join's projection takes `(k, (l, r))` and a half join's takes `(k, ((l, r), t))`. The recipes are rewritten to those forms, and their transcripts are unchanged. The reference gives `join(other)` and `half_join(other, strict)` as the reference forms, with a projection as sugar for a `map`. The `sum` recipe and the first program use `| sum`, and the reductions table lists it with its expansion. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5
frankmcsherry
added a commit
that referenced
this pull request
Nov 21, 2026
Since #1103 and #1104 a join's projection takes `(k, (l, r))` and a half join's takes `(k, ((l, r), t))`. The recipes are rewritten to those forms, and their transcripts are unchanged. The reference gives `join(other)` and `half_join(other, strict)` as the reference forms, with a projection as sugar for a `map`. The `sum` recipe and the first program use `| sum`, and the reductions table lists it with its expansion. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5
frankmcsherry
added a commit
that referenced
this pull request
Nov 21, 2026
Since #1103 and #1104 a join's projection takes `(k, (l, r))` and a half join's takes `(k, ((l, r), t))`. The recipes are rewritten to those forms, and their transcripts are unchanged. The reference gives `join(other)` and `half_join(other, strict)` as the reference forms, with a projection as sugar for a `map`. The `sum` recipe and the first program use `| sum`, and the reductions table lists it with its expansion. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5
frankmcsherry
added a commit
that referenced
this pull request
Nov 21, 2026
* DDIR: recipes, one checked program per idiom The user study found that the idioms that make DDIR programs fast or correct are hard to discover. `examples/recipes` holds one small program for each, with comments that explain the idiom, a session that runs it on a few rows, and the session's recorded transcript. `server/tests/recipes.rs` pipes each session into a server and checks its transcript at one and three workers. The first eight: - `sum`: an integer sum per key, via `join_function` into the diff and `count`. - `top_k`: top k through a tree of `reduce(limit k)` groups. - `retention`: a sliding window, retracting each row with `lift` and `delay`. - `upsert`: a keyed table with put and delete, held in a root feedback variable. - `left_join`: an outer join, with a declared enum for the missing side. - `connected_components`: label propagation, prioritized with `delay`. - `shortest_paths`: delta stepping, with buckets spaced in rounds. - `triangles`: a delta join with `lift` and `half_join`. The DDIR README links to the recipes' README. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5 * DDIR: a draft language reference `LANGUAGE.md` gathers what was spread across the `pipe.rs` doc comment, corgi's op table and the README: collections and shapes, statements and sources, every collection operator, join fragments, lambdas and the scalar operations, time and scopes, iteration including root variables, and the language's design positions. Each idiom links to the recipe that shows it. The fragment doc comment in `pipe.rs` still said a one-variable fragment is keyed by `(x)`; since #1096 it is keyed by the variable itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5 * DDIR reference: operators by reference form and sugar Each operator group names its reference form, and each sugar shows its expansion. join_function is the reference for linear operators, join for joins, and reduce for reductions. lift, lift_iter and half_join move to their own group, as they read times and are not functions of data alone. Every expansion was run against its sugar on the server, with equal results. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5 * DDIR reference: count drops values, no value or lift_iter This follows the ddir-prune branch, which removes value and lift_iter and has count project its values away. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5 * DDIR reference: inputs by label, bound at load; no import Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5 * DDIR recipes and reference: joins pass the joined row; use sum Since #1103 and #1104 a join's projection takes `(k, (l, r))` and a half join's takes `(k, ((l, r), t))`. The recipes are rewritten to those forms, and their transcripts are unchanged. The reference gives `join(other)` and `half_join(other, strict)` as the reference forms, with a projection as sugar for a `map`. The `sum` recipe and the first program use `| sum`, and the reductions table lists it with its expansion. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5 --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A DDIR equijoin can now be written without a projection. For matching inputs
(k, l)and(k, r),left | join(right)produces(k, (l, r)). The optional projection receives that same row, so these expressions compute the same collection.This changes existing projected joins. Migrate a binder such as
(k, l, r) -> bodyto(k, (l, r)) -> body. The projection still runs inside the join operator. The examples, benchmark generators, and reviewed plan snapshots use the new binder.A regression test checks the bare join, an explicit identity, and both forms of a rekeying projection on nested values across inserts and a retraction. The interactive and server test suites pass. Two regenerations reproduced every plan snapshot byte for byte, and all 33 Advent of Code programs passed planning.