Skip to content

DDIR: a language reference and recipes - #1106

Merged
frankmcsherry merged 6 commits into
master-nextfrom
ddir-recipes
Nov 21, 2026
Merged

frankmcsherry merged 6 commits into
master-nextfrom
ddir-recipes

Conversation

@frankmcsherry

@frankmcsherry frankmcsherry commented Nov 21, 2026 •

Copy link
Copy Markdown
Member

A language reference, interactive/LANGUAGE.md, and interactive/examples/recipes, one checked program per idiom.
The reference describes inputs by label, with no import, as #1105 made them.

The reference

It covers what a program is made of, every collection operator, the scalar language inside operators, time and scopes, and the positions the language takes on purpose (no NULL, nothing durable, one trust domain, and so on).
Each group of operators names one reference form, and the rest of the group is sugar for it, with the expansion:

  • linear operators: join_function, with map, filter, flatmap, negate and delay as sugar (+ is its own operator);
  • joins: join(other), with a projection and join fragments as sugar;
  • reductions: reduce, with min, count, sum, distinct and collect as sugar;
  • operators that read time, kept apart: lift and half_join.

Every expansion was checked on a server against its sugar.

The recipes

Eight programs, each with comments that explain the idiom, a session that runs it on a few rows, and the session's recorded transcript: sum, top_k, retention, upsert, left_join, connected_components, shortest_paths and triangles.
server/tests/recipes.rs pipes each session into a server and checks its transcript at one and three workers.
The recipes use the joined-row forms of #1103 and #1104, and | sum from #1102.

The reference is meant to be read for what to clean up next, and updated as the language changes.

🤖 Generated with Claude Code

https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5

@frankmcsherry
frankmcsherry force-pushed the ddir-recipes branch 4 times, most recently from fb22734 to 4f52fd8 Compare November 21, 2026 03:21
frankmcsherry and others added 6 commits November 20, 2026 22:34
The user study found that the idioms that make DDIR programs fast or correct are hard to discover.
`examples/recipes` holds one small program for each, with comments that explain the idiom, a session that runs it on a few rows, and the session's recorded transcript.
`server/tests/recipes.rs` pipes each session into a server and checks its transcript at one and three workers.

The first eight:
- `sum`: an integer sum per key, via `join_function` into the diff and `count`.
- `top_k`: top k through a tree of `reduce(limit k)` groups.
- `retention`: a sliding window, retracting each row with `lift` and `delay`.
- `upsert`: a keyed table with put and delete, held in a root feedback variable.
- `left_join`: an outer join, with a declared enum for the missing side.
- `connected_components`: label propagation, prioritized with `delay`.
- `shortest_paths`: delta stepping, with buckets spaced in rounds.
- `triangles`: a delta join with `lift` and `half_join`.

The DDIR README links to the recipes' README.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5
`LANGUAGE.md` gathers what was spread across the `pipe.rs` doc comment, corgi's op table and the README: collections and shapes, statements and sources, every collection operator, join fragments, lambdas and the scalar operations, time and scopes, iteration including root variables, and the language's design positions.
Each idiom links to the recipe that shows it.

The fragment doc comment in `pipe.rs` still said a one-variable fragment is keyed by `(x)`; since #1096 it is keyed by the variable itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5
Each operator group names its reference form, and each sugar shows its expansion.
join_function is the reference for linear operators, join for joins, and reduce for reductions.
lift, lift_iter and half_join move to their own group, as they read times and are not functions of data alone.
Every expansion was run against its sugar on the server, with equal results.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5
This follows the ddir-prune branch, which removes value and lift_iter and has count project its values away.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5
Since #1103 and #1104 a join's projection takes `(k, (l, r))` and a half join's takes `(k, ((l, r), t))`.
The recipes are rewritten to those forms, and their transcripts are unchanged.
The reference gives `join(other)` and `half_join(other, strict)` as the reference forms, with a projection as sugar for a `map`.
The `sum` recipe and the first program use `| sum`, and the reductions table lists it with its expansion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5
@frankmcsherry
frankmcsherry changed the base branch from ddir-inputs to master-next November 21, 2026 03:35
@frankmcsherry
frankmcsherry merged commit 92c9ae1 into master-next Nov 21, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant