Repository navigation
DDIR: a language reference and recipes - #1106
Merged
Merged
Conversation
frankmcsherry
force-pushed
the
ddir-recipes
branch
4 times, most recently
from
November 21, 2026 03:21
fb22734 to
4f52fd8
Compare
The user study found that the idioms that make DDIR programs fast or correct are hard to discover. `examples/recipes` holds one small program for each, with comments that explain the idiom, a session that runs it on a few rows, and the session's recorded transcript. `server/tests/recipes.rs` pipes each session into a server and checks its transcript at one and three workers. The first eight: - `sum`: an integer sum per key, via `join_function` into the diff and `count`. - `top_k`: top k through a tree of `reduce(limit k)` groups. - `retention`: a sliding window, retracting each row with `lift` and `delay`. - `upsert`: a keyed table with put and delete, held in a root feedback variable. - `left_join`: an outer join, with a declared enum for the missing side. - `connected_components`: label propagation, prioritized with `delay`. - `shortest_paths`: delta stepping, with buckets spaced in rounds. - `triangles`: a delta join with `lift` and `half_join`. The DDIR README links to the recipes' README. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5
`LANGUAGE.md` gathers what was spread across the `pipe.rs` doc comment, corgi's op table and the README: collections and shapes, statements and sources, every collection operator, join fragments, lambdas and the scalar operations, time and scopes, iteration including root variables, and the language's design positions. Each idiom links to the recipe that shows it. The fragment doc comment in `pipe.rs` still said a one-variable fragment is keyed by `(x)`; since #1096 it is keyed by the variable itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5
Each operator group names its reference form, and each sugar shows its expansion. join_function is the reference for linear operators, join for joins, and reduce for reductions. lift, lift_iter and half_join move to their own group, as they read times and are not functions of data alone. Every expansion was run against its sugar on the server, with equal results. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5
This follows the ddir-prune branch, which removes value and lift_iter and has count project its values away. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5
Since #1103 and #1104 a join's projection takes `(k, (l, r))` and a half join's takes `(k, ((l, r), t))`. The recipes are rewritten to those forms, and their transcripts are unchanged. The reference gives `join(other)` and `half_join(other, strict)` as the reference forms, with a projection as sugar for a `map`. The `sum` recipe and the first program use `| sum`, and the reductions table lists it with its expansion. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5
frankmcsherry
force-pushed
the
ddir-recipes
branch
from
November 21, 2026 03:35
4f52fd8 to
e53f801
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A language reference,
interactive/LANGUAGE.md, andinteractive/examples/recipes, one checked program per idiom.The reference describes inputs by label, with no
import, as #1105 made them.The reference
It covers what a program is made of, every collection operator, the scalar language inside operators, time and scopes, and the positions the language takes on purpose (no NULL, nothing durable, one trust domain, and so on).
Each group of operators names one reference form, and the rest of the group is sugar for it, with the expansion:
join_function, withmap,filter,flatmap,negateanddelayas sugar (+is its own operator);join(other), with a projection and join fragments as sugar;reduce, withmin,count,sum,distinctandcollectas sugar;liftandhalf_join.Every expansion was checked on a server against its sugar.
The recipes
Eight programs, each with comments that explain the idiom, a session that runs it on a few rows, and the session's recorded transcript:
sum,top_k,retention,upsert,left_join,connected_components,shortest_pathsandtriangles.server/tests/recipes.rspipes each session into a server and checks its transcript at one and three workers.The recipes use the joined-row forms of #1103 and #1104, and
| sumfrom #1102.The reference is meant to be read for what to clean up next, and updated as the language changes.
🤖 Generated with Claude Code
https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5