From 54eb6ebb379996f72b908d58988a57de38ca05e3 Mon Sep 17 00:00:00 2001 From: Frank McSherry Date: Thu, 19 Nov 2026 21:49:48 -0500 Subject: [PATCH 1/6] DDIR: recipes, one checked program per idiom The user study found that the idioms that make DDIR programs fast or correct are hard to discover. `examples/recipes` holds one small program for each, with comments that explain the idiom, a session that runs it on a few rows, and the session's recorded transcript. `server/tests/recipes.rs` pipes each session into a server and checks its transcript at one and three workers. The first eight: - `sum`: an integer sum per key, via `join_function` into the diff and `count`. - `top_k`: top k through a tree of `reduce(limit k)` groups. - `retention`: a sliding window, retracting each row with `lift` and `delay`. - `upsert`: a keyed table with put and delete, held in a root feedback variable. - `left_join`: an outer join, with a declared enum for the missing side. - `connected_components`: label propagation, prioritized with `delay`. - `shortest_paths`: delta stepping, with buckets spaced in rounds. - `triangles`: a delta join with `lift` and `half_join`. The DDIR README links to the recipes' README. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5 --- interactive/README.md | 2 +- interactive/examples/recipes/README.md | 30 +++++++++ .../examples/recipes/connected_components.ddp | 22 +++++++ .../examples/recipes/connected_components.out | 27 ++++++++ .../examples/recipes/connected_components.txt | 21 ++++++ interactive/examples/recipes/left_join.ddp | 17 +++++ interactive/examples/recipes/left_join.out | 16 +++++ interactive/examples/recipes/left_join.txt | 19 ++++++ interactive/examples/recipes/retention.ddp | 13 ++++ interactive/examples/recipes/retention.out | 16 +++++ interactive/examples/recipes/retention.txt | 17 +++++ .../examples/recipes/shortest_paths.ddp | 22 +++++++ .../examples/recipes/shortest_paths.out | 23 +++++++ .../examples/recipes/shortest_paths.txt | 21 ++++++ interactive/examples/recipes/sum.ddp | 13 ++++ interactive/examples/recipes/sum.out | 15 +++++ interactive/examples/recipes/sum.txt | 18 +++++ interactive/examples/recipes/top_k.ddp | 19 ++++++ interactive/examples/recipes/top_k.out | 21 ++++++ interactive/examples/recipes/top_k.txt | 22 +++++++ interactive/examples/recipes/triangles.ddp | 40 +++++++++++ interactive/examples/recipes/triangles.out | 24 +++++++ interactive/examples/recipes/triangles.txt | 22 +++++++ interactive/examples/recipes/upsert.ddp | 25 +++++++ interactive/examples/recipes/upsert.out | 33 ++++++++++ interactive/examples/recipes/upsert.txt | 34 ++++++++++ interactive/server/tests/recipes.rs | 66 +++++++++++++++++++ 27 files changed, 617 insertions(+), 1 deletion(-) create mode 100644 interactive/examples/recipes/README.md create mode 100644 interactive/examples/recipes/connected_components.ddp create mode 100644 interactive/examples/recipes/connected_components.out create mode 100644 interactive/examples/recipes/connected_components.txt create mode 100644 interactive/examples/recipes/left_join.ddp create mode 100644 interactive/examples/recipes/left_join.out create mode 100644 interactive/examples/recipes/left_join.txt create mode 100644 interactive/examples/recipes/retention.ddp create mode 100644 interactive/examples/recipes/retention.out create mode 100644 interactive/examples/recipes/retention.txt create mode 100644 interactive/examples/recipes/shortest_paths.ddp create mode 100644 interactive/examples/recipes/shortest_paths.out create mode 100644 interactive/examples/recipes/shortest_paths.txt create mode 100644 interactive/examples/recipes/sum.ddp create mode 100644 interactive/examples/recipes/sum.out create mode 100644 interactive/examples/recipes/sum.txt create mode 100644 interactive/examples/recipes/top_k.ddp create mode 100644 interactive/examples/recipes/top_k.out create mode 100644 interactive/examples/recipes/top_k.txt create mode 100644 interactive/examples/recipes/triangles.ddp create mode 100644 interactive/examples/recipes/triangles.out create mode 100644 interactive/examples/recipes/triangles.txt create mode 100644 interactive/examples/recipes/upsert.ddp create mode 100644 interactive/examples/recipes/upsert.out create mode 100644 interactive/examples/recipes/upsert.txt create mode 100644 interactive/server/tests/recipes.rs diff --git a/interactive/README.md b/interactive/README.md index 7057a2766..e1fd09581 100644 --- a/interactive/README.md +++ b/interactive/README.md @@ -1,6 +1,6 @@ # DDIR -[Server command reference](server/README.md#server-command-reference) +[Server command reference](server/README.md#server-command-reference) · [Recipes](examples/recipes/README.md) An interpreted language (and runtime) for computation over continually changing collections. diff --git a/interactive/examples/recipes/README.md b/interactive/examples/recipes/README.md new file mode 100644 index 000000000..9947fcd2a --- /dev/null +++ b/interactive/examples/recipes/README.md @@ -0,0 +1,30 @@ +# Recipes + +Each recipe is a small program for one idiom, with a session that runs it on a few rows. +The idioms are the ones that make DDIR programs fast or correct, and that are hard to find from the operator list alone. + +Run a recipe from `interactive/`, and compare with its recorded transcript: + +```sh +cargo run --release -p ddir-server < examples/recipes/upsert.txt +``` + +`NAME.ddp` is the program and explains the idiom in its comments. +`NAME.txt` feeds it, seals its inputs, and peeks the results. +`NAME.out` is the transcript, checked at one and three workers by `server/tests/recipes.rs`. +After an intended change, `UPDATE_RECIPES=1 cargo test --release -p ddir-server --test recipes` records the transcripts again. + +| Recipe | Idiom | Use it for | +|---|---|---| +| [`sum`](sum.ddp) | `join_function` moves a value into the diff, and `count` adds up the diffs | sums of integers, incrementally | +| [`top_k`](top_k.ddp) | a tree of `reduce(limit k)` groups | top k over many rows, with small updates | +| [`retention`](retention.ddp) | `lift` and `delay` retract each row a fixed time after it arrives | sliding windows, expiry | +| [`upsert`](upsert.ddp) | a root `var` holds the table one time ago; writes count at their own time only | a keyed table with put and delete | +| [`left_join`](left_join.ddp) | the unmatched rows are the input minus the matched ones; a declared `enum` for the missing value | outer joins, without NULL | +| [`connected_components`](connected_components.ddp) | `delay` holds back large labels, so small ones spread first | label propagation, prioritized iteration | +| [`shortest_paths`](shortest_paths.ddp) | `delay` to bucket `d / Δ`, spaced several rounds apart | weighted shortest paths, delta stepping | +| [`triangles`](triangles.ddp) | `lift` and `half_join` along each use of a relation | multiway joins without large intermediate results | + +Two facts the recipes lean on: +- **Inputs are multisets.** A row's count is the sum of its diffs, so a second put of a key adds a second row; replacing a value is a program's job, as in `upsert`. +- **`(e)` is a one-field tuple, not `e`.** Parentheses build a tuple. Write `t` where you mean the value itself; `(t)` is a different shape. diff --git a/interactive/examples/recipes/connected_components.ddp b/interactive/examples/recipes/connected_components.ddp new file mode 100644 index 000000000..879046e64 --- /dev/null +++ b/interactive/examples/recipes/connected_components.ddp @@ -0,0 +1,22 @@ +-- Connected components by label propagation, with prioritized iteration. +-- +-- Each node starts labeled by itself, and repeatedly takes the least label among its neighbors. +-- Unprioritized, every node's own label starts spreading in the first round, and most of that +-- work is undone when a smaller label arrives. Here `delay` holds node n's own label back to round +-- 256 * bit_length(n), so small labels spread before large ones enter; `scc.ddp` does the same. +-- On a 1M-edge graph this took the first computation from 1.39 s to 0.55 s. +-- The cost moves to some later edits: deleting and re-adding a bridge to a long path took +-- longer with the delay than without (0.7 s and 3.4 s against 37 ms, for a 300-node path). + +let raw = input 0 : (int, int); -- an undirected edge, either way round +let edges = raw + (raw | map((a, b) -> (b, a))) | distinct; +let nodes = edges | map((a, _) -> (a, ())) | distinct; + +cc: { + -- The inner time is (epoch, round): `0` leaves the epoch alone and sets the round. + let seeds = nodes | map((n, _) -> (n, n)) | delay((_, n) -> (0, (256, n call bit_length) mul)); + let proposals = labels | join(edges, (_, label, dst) -> (dst, label)); + set labels = seeds + proposals | min; +} + +output "labels" = cc::labels; diff --git a/interactive/examples/recipes/connected_components.out b/interactive/examples/recipes/connected_components.out new file mode 100644 index 000000000..11d84be08 --- /dev/null +++ b/interactive/examples/recipes/connected_components.out @@ -0,0 +1,27 @@ +_1 ok installed "cc" +_2 ok fed "cc" input 0 at t=0 +_3 ok fed "cc" input 0 at t=0 +_4 ok fed "cc" input 0 at t=0 +_5 ok sealed "cc" input 0 to 1 +_6 data 1 val=1 +_6 data 2 val=1 +_6 data 3 val=1 +_6 data 4 val=4 +_6 data 5 val=4 +_6 ok as-of=0 +_7 ok fed "cc" input 0 at t=1 +_8 ok sealed "cc" input 0 to 2 +_9 data 1 val=1 +_9 data 2 val=1 +_9 data 3 val=1 +_9 data 4 val=1 +_9 data 5 val=1 +_9 ok as-of=1 +_10 ok fed "cc" input 0 at t=2 +_11 ok sealed "cc" input 0 to 3 +_12 data 2 val=2 +_12 data 3 val=2 +_12 data 4 val=2 +_12 data 5 val=2 +_12 ok as-of=2 +_13 ok bye diff --git a/interactive/examples/recipes/connected_components.txt b/interactive/examples/recipes/connected_components.txt new file mode 100644 index 000000000..90ed17f3d --- /dev/null +++ b/interactive/examples/recipes/connected_components.txt @@ -0,0 +1,21 @@ +# Connected components with prioritized label propagation: see connected_components.ddp. +# Run from interactive/: ddir_server < examples/recipes/connected_components.txt +load cc from examples/recipes/connected_components.ddp + +# Time 0: two components, {1, 2, 3} and {4, 5}. +feed cc 0 1 val=2 time=0 +feed cc 0 3 val=2 time=0 +feed cc 0 5 val=4 time=0 +seal cc 0 1 +peek cc::labels + +# Time 1: the edge 3-4 joins them. +feed cc 0 3 val=4 time=1 +seal cc 0 2 +peek cc::labels + +# Time 2: removing 1-2 leaves node 1 with no edges, so no label; the rest take label 2. +feed cc 0 1 val=2 time=2 diff=-1 +seal cc 0 3 +peek cc::labels +exit diff --git a/interactive/examples/recipes/left_join.ddp b/interactive/examples/recipes/left_join.ddp new file mode 100644 index 000000000..9e0ae2643 --- /dev/null +++ b/interactive/examples/recipes/left_join.ddp @@ -0,0 +1,17 @@ +-- A left join: every order, with its customer's name when there is one. +-- +-- There is no NULL; a missing value is a variant. Declaring the enum gives each variant its +-- payload shape, so `() inject Unknown` needs nothing more. (The built-in `Option` cannot yet +-- build `None` this way: its `Some` lane has no shape until a value supplies one.) +-- The unmatched rows are the input minus the ones the join matched. + +enum Name = Known List(int) | Unknown; + +let orders = input 0 : (int, int); -- (order, customer) +let customers = input 1 : (int, List(int)); -- (customer, name); a string is a list of bytes + +let by_customer = orders | map((order, customer) -> (customer, order)); +let matched = by_customer | join(customers, (c, order, name) -> (order, (c, name inject Known))); +let unmatched = by_customer - (by_customer | join(customers, (c, order, _) -> (c, order))); + +output "orders" = matched + (unmatched | map((c, order) -> (order, (c, () inject Unknown)))); diff --git a/interactive/examples/recipes/left_join.out b/interactive/examples/recipes/left_join.out new file mode 100644 index 000000000..94d55615e --- /dev/null +++ b/interactive/examples/recipes/left_join.out @@ -0,0 +1,16 @@ +_1 ok installed "lj" +_2 ok fed "lj" input 1 at t=0 +_3 ok fed "lj" input 0 at t=0 +_4 ok fed "lj" input 0 at t=0 +_5 ok sealed "lj" input 0 to 1 +_6 ok sealed "lj" input 1 to 1 +_7 data 100 val=tuple(1,inject(0,list(97,110,110))) +_7 data 101 val=tuple(2,inject(1,tuple())) +_7 ok as-of=0 +_8 ok fed "lj" input 1 at t=1 +_9 ok sealed "lj" input 0 to 2 +_10 ok sealed "lj" input 1 to 2 +_11 data 100 val=tuple(1,inject(0,list(97,110,110))) +_11 data 101 val=tuple(2,inject(0,list(98,111))) +_11 ok as-of=1 +_12 ok bye diff --git a/interactive/examples/recipes/left_join.txt b/interactive/examples/recipes/left_join.txt new file mode 100644 index 000000000..27bab4ffb --- /dev/null +++ b/interactive/examples/recipes/left_join.txt @@ -0,0 +1,19 @@ +# A left join with a declared enum for the missing side: see left_join.ddp. +# Run from interactive/: ddir_server < examples/recipes/left_join.txt +load lj from examples/recipes/left_join.ddp + +# A feed spells a string as its bytes: "ann" is list(97,110,110). +feed lj 1 1 val=list(97,110,110) time=0 +feed lj 0 100 val=1 time=0 +feed lj 0 101 val=2 time=0 +seal lj 0 1 +seal lj 1 1 +# Order 101's customer is unknown: `inject(1,tuple())`. Order 100's is `inject(0,)`. +peek lj::orders + +# Customer 2 arrives, and order 101 moves from Unknown to Known. +feed lj 1 2 val=list(98,111) time=1 +seal lj 0 2 +seal lj 1 2 +peek lj::orders +exit diff --git a/interactive/examples/recipes/retention.ddp b/interactive/examples/recipes/retention.ddp new file mode 100644 index 000000000..636ba58a4 --- /dev/null +++ b/interactive/examples/recipes/retention.ddp @@ -0,0 +1,13 @@ +-- A sliding window: count each key's events over the last 3 times. +-- +-- `lift` pairs each row with its time t, and `delay` moves a copy of the row to t + 3. +-- Subtracting that copy retracts each event 3 times after it arrived, so the window holds only +-- recent events, and its state stays the size of the window rather than of the whole history. +-- Joining against a clock input instead would redo the work for every event at every tick. + +let events = input 0 : (int, ()); -- an event for a key, at the time it is fed + +let expired = events | lift | delay((_, (_, (t))) -> (t, 3) add) | map((k, _) -> (k, ())); +let recent = events - expired; + +output "recent" = recent | count; diff --git a/interactive/examples/recipes/retention.out b/interactive/examples/recipes/retention.out new file mode 100644 index 000000000..c440ea399 --- /dev/null +++ b/interactive/examples/recipes/retention.out @@ -0,0 +1,16 @@ +_1 ok installed "win" +_2 ok fed "win" input 0 at t=0 +_3 ok fed "win" input 0 at t=1 +_4 ok fed "win" input 0 at t=1 +_5 ok fed "win" input 0 at t=2 +_6 ok sealed "win" input 0 to 3 +_7 data 7 val=3 +_7 data 8 val=1 +_7 ok as-of=2 +_8 ok sealed "win" input 0 to 5 +_9 data 7 val=2 +_9 data 8 val=1 +_9 ok as-of=3 +_10 data 7 val=1 +_10 ok as-of=4 +_11 ok bye diff --git a/interactive/examples/recipes/retention.txt b/interactive/examples/recipes/retention.txt new file mode 100644 index 000000000..b3bfdedcf --- /dev/null +++ b/interactive/examples/recipes/retention.txt @@ -0,0 +1,17 @@ +# A sliding window with `delay`: see retention.ddp. +# Run from interactive/: ddir_server < examples/recipes/retention.txt +load win from examples/recipes/retention.ddp + +feed win 0 7 time=0 +feed win 0 7 time=1 +feed win 0 8 time=1 +feed win 0 7 time=2 +seal win 0 3 +# Times 0, 1 and 2 are all in the window at time 2. +peek win::recent + +# At time 3 the event from time 0 leaves the window; at time 4, those from time 1 leave. +seal win 0 5 +peek win::recent as-of 3 +peek win::recent as-of 4 +exit diff --git a/interactive/examples/recipes/shortest_paths.ddp b/interactive/examples/recipes/shortest_paths.ddp new file mode 100644 index 000000000..8d20d543c --- /dev/null +++ b/interactive/examples/recipes/shortest_paths.ddp @@ -0,0 +1,22 @@ +-- Single-source shortest paths with delta stepping. +-- +-- Plain relaxation in a loop finds every path, but on weighted graphs it keeps improving long +-- tentative distances that a shorter path later replaces. Delta stepping settles distances in +-- buckets of width Δ, nearest first. Here `delay` puts a proposed distance d at round +-- (d / 64) * 16: bucket d / 64, spaced 16 rounds apart, so a bucket's hop-by-hop relaxations +-- finish before the next bucket starts. With the spacing at 1 a bucket's relaxations run into +-- the next bucket's and the delay gains nothing. +-- On a 500 x 500 grid with weights 1 to 100 this took the first computation from 64.5 s to 0.9 s. +-- Choose Δ near the typical edge weight, and the spacing above the hops a bucket needs. + +let edges = input 0 : (int, (int, int)); -- (src, (dst, weight)) +let roots = input 1 : (int, ()); -- the sources, at distance 0 + +sssp: { + let proposals = dist + | join(edges, (_, d, (dst, w)) -> (dst, (d, w) add)) + | delay((_, d) -> (0, ((d, 64) div, 16) mul)); + set dist = (roots | map((n, ()) -> (n, 0))) + proposals | min; +} + +output "dist" = sssp::dist; diff --git a/interactive/examples/recipes/shortest_paths.out b/interactive/examples/recipes/shortest_paths.out new file mode 100644 index 000000000..246a482f5 --- /dev/null +++ b/interactive/examples/recipes/shortest_paths.out @@ -0,0 +1,23 @@ +_1 ok installed "sp" +_2 ok fed "sp" input 0 at t=0 +_3 ok fed "sp" input 0 at t=0 +_4 ok fed "sp" input 0 at t=0 +_5 ok fed "sp" input 0 at t=0 +_6 ok fed "sp" input 1 at t=0 +_7 ok sealed "sp" input 0 to 1 +_8 ok sealed "sp" input 1 to 1 +_9 data 1 val=0 +_9 data 2 val=200 +_9 data 3 val=400 +_9 data 4 val=410 +_9 ok as-of=0 +_10 ok fed "sp" input 0 at t=1 +_11 ok fed "sp" input 0 at t=1 +_12 ok sealed "sp" input 0 to 2 +_13 ok sealed "sp" input 1 to 2 +_14 data 1 val=0 +_14 data 2 val=200 +_14 data 3 val=500 +_14 data 4 val=510 +_14 ok as-of=1 +_15 ok bye diff --git a/interactive/examples/recipes/shortest_paths.txt b/interactive/examples/recipes/shortest_paths.txt new file mode 100644 index 000000000..2ec40885e --- /dev/null +++ b/interactive/examples/recipes/shortest_paths.txt @@ -0,0 +1,21 @@ +# Shortest paths with delta stepping: see shortest_paths.ddp. +# Run from interactive/: ddir_server < examples/recipes/shortest_paths.txt +load sp from examples/recipes/shortest_paths.ddp + +# Time 0: 1 -> 2 -> 3 costs 200 + 200; the direct edge 1 -> 3 costs 500. +feed sp 0 1 val=tuple(2,200) time=0 +feed sp 0 2 val=tuple(3,200) time=0 +feed sp 0 1 val=tuple(3,500) time=0 +feed sp 0 3 val=tuple(4,10) time=0 +feed sp 1 1 time=0 +seal sp 0 1 +seal sp 1 1 +peek sp::dist + +# Time 1: the edge 2 -> 3 gets dearer, and the direct edge wins. +feed sp 0 2 val=tuple(3,200) time=1 diff=-1 +feed sp 0 2 val=tuple(3,900) time=1 +seal sp 0 2 +seal sp 1 2 +peek sp::dist +exit diff --git a/interactive/examples/recipes/sum.ddp b/interactive/examples/recipes/sum.ddp new file mode 100644 index 000000000..bd94838a5 --- /dev/null +++ b/interactive/examples/recipes/sum.ddp @@ -0,0 +1,13 @@ +-- Sum an integer per key: move the value into the diff, and `count` adds up the diffs. +-- +-- `join_function` replaces each row with a list of `((key, val), time, diff)`. +-- Here each sale becomes one row `(store, ())` at its own time (`0` leaves the time as it is), +-- with the amount as its diff; `count` then gives each store the total of its diffs. +-- The sum is maintained incrementally, where folding a list of values would rebuild it per change. +-- A total of zero has no row, like any record whose diffs cancel; it is bounded by `i64`. + +let sales = input 0 : (int, int); -- (store, amount); a refund is a negative amount + +output "total" = sales + | join_function((store, amount) -> ((store, ()), 0, amount) enlist) + | count; diff --git a/interactive/examples/recipes/sum.out b/interactive/examples/recipes/sum.out new file mode 100644 index 000000000..1b660ce9e --- /dev/null +++ b/interactive/examples/recipes/sum.out @@ -0,0 +1,15 @@ +_1 ok installed "sum" +_2 ok fed "sum" input 0 at t=0 +_3 ok fed "sum" input 0 at t=0 +_4 ok fed "sum" input 0 at t=0 +_5 ok fed "sum" input 0 at t=0 +_6 ok sealed "sum" input 0 to 1 +_7 data 1 val=19 +_7 data 2 val=40 +_7 ok as-of=0 +_8 ok fed "sum" input 0 at t=1 +_9 ok fed "sum" input 0 at t=1 +_10 ok sealed "sum" input 0 to 2 +_11 data 1 val=20 +_11 ok as-of=1 +_12 ok bye diff --git a/interactive/examples/recipes/sum.txt b/interactive/examples/recipes/sum.txt new file mode 100644 index 000000000..4d58649a2 --- /dev/null +++ b/interactive/examples/recipes/sum.txt @@ -0,0 +1,18 @@ +# Sum per key with `count`: see sum.ddp. +# Run from interactive/: ddir_server < examples/recipes/sum.txt +load sum from examples/recipes/sum.ddp + +# Time 0: three sales at store 1, one at store 2. +feed sum 0 1 val=5 time=0 +feed sum 0 1 val=7 time=0 +feed sum 0 1 val=7 time=0 +feed sum 0 2 val=40 time=0 +seal sum 0 1 +peek sum::total + +# Time 1: a refund at store 2 brings it to zero, and its row goes away. +feed sum 0 2 val=-40 time=1 +feed sum 0 1 val=1 time=1 +seal sum 0 2 +peek sum::total +exit diff --git a/interactive/examples/recipes/top_k.ddp b/interactive/examples/recipes/top_k.ddp new file mode 100644 index 000000000..c4589c8e3 --- /dev/null +++ b/interactive/examples/recipes/top_k.ddp @@ -0,0 +1,19 @@ +-- The top 3 scores overall, maintained as scores change. +-- +-- `reduce(limit 3, ...)` hands its lambda only a key's first 3 values in value order, so storing +-- the negated score puts the highest first. One group holding every player would redo that +-- group's work on every update; a tree of groups keeps each update small. Here players fall into +-- 64 buckets, each bucket keeps its own top 3, and the root takes the top 3 of those. +-- An update then touches one bucket and the 64 * 3 candidates at the root. +-- Ties order by player, the second field of the value. + +let scores = input 0 : (int, int); -- (player, score) + +let buckets = scores + | map((player, score) -> ((player, 64) rem, (score neg, player))) + | reduce(limit 3, (_, vals) -> vals); +let top = buckets + | map((_, v) -> ((), v)) + | reduce(limit 3, (_, vals) -> vals); + +output "top" = top | map((_, (negated, player)) -> (player, negated neg)); diff --git a/interactive/examples/recipes/top_k.out b/interactive/examples/recipes/top_k.out new file mode 100644 index 000000000..2222a08a3 --- /dev/null +++ b/interactive/examples/recipes/top_k.out @@ -0,0 +1,21 @@ +_1 ok installed "top" +_2 ok fed "top" input 0 at t=0 +_3 ok fed "top" input 0 at t=0 +_4 ok fed "top" input 0 at t=0 +_5 ok fed "top" input 0 at t=0 +_6 ok fed "top" input 0 at t=0 +_7 ok sealed "top" input 0 to 1 +_8 data 2 val=80 +_8 data 65 val=70 +_8 data 129 val=90 +_8 ok as-of=0 +_9 ok fed "top" input 0 at t=1 +_10 ok fed "top" input 0 at t=1 +_11 ok fed "top" input 0 at t=1 +_12 ok fed "top" input 0 at t=1 +_13 ok sealed "top" input 0 to 2 +_14 data 2 val=80 +_14 data 3 val=95 +_14 data 65 val=70 +_14 ok as-of=1 +_15 ok bye diff --git a/interactive/examples/recipes/top_k.txt b/interactive/examples/recipes/top_k.txt new file mode 100644 index 000000000..ef7b655c7 --- /dev/null +++ b/interactive/examples/recipes/top_k.txt @@ -0,0 +1,22 @@ +# Top k through a tree of `reduce(limit k)`: see top_k.ddp. +# Run from interactive/: ddir_server < examples/recipes/top_k.txt +load top from examples/recipes/top_k.ddp + +# Time 0: five players. +feed top 0 1 val=50 time=0 +feed top 0 2 val=80 time=0 +feed top 0 65 val=70 time=0 +feed top 0 3 val=10 time=0 +feed top 0 129 val=90 time=0 +seal top 0 1 +peek top::top + +# Time 1: player 3 climbs to the top, and player 129 drops out: a score change is a retraction +# of the old row and an insertion of the new one. +feed top 0 3 val=10 time=1 diff=-1 +feed top 0 3 val=95 time=1 +feed top 0 129 val=90 time=1 diff=-1 +feed top 0 129 val=20 time=1 +seal top 0 2 +peek top::top +exit diff --git a/interactive/examples/recipes/triangles.ddp b/interactive/examples/recipes/triangles.ddp new file mode 100644 index 000000000..825943d38 --- /dev/null +++ b/interactive/examples/recipes/triangles.ddp @@ -0,0 +1,40 @@ +-- Triangle counting as a delta join: no collection of wedges is ever built. +-- +-- Triangles are T(a, b, c) :- E(a, b), E(a, c), E(b, c), with each edge pointed from its larger +-- end to its smaller, and b > c. The plain plan joins E with itself into every wedge (a path of +-- two edges) and arranges them all: 60.8M wedges for a 1M-edge graph. A delta join instead +-- handles each change to E once per use of E in the rule, and joins that change against the +-- other two uses with `half_join`, which reads the arranged edges as of the change's time. +-- For the use numbered i, a change meets later uses at or before its time (strict 0) and earlier +-- uses strictly before it (strict 1), so a change that touches two uses counts once. +-- On the 1M-edge graph a single edge update went from 2.86 ms to 0.25 ms. + +let raw = input 0 : (int, int); -- an undirected edge, either way round +let down = raw + | filter((a, b) -> (a, b) ne) + | map((a, b) -> ((((a, b) gt, a, b) select, ((a, b) gt, b, a) select), ())) + | distinct; +let out = down | map(((a, b), ()) -> (a, b)); -- keyed by the larger end +let rev = down | map(((a, b), ()) -> (b, a)); -- keyed by the smaller end +let pairs = down | arrange; -- keyed by the whole edge + +-- A change to E(a, b): find E(a, c), then check E(b, c). +let p1 = out | lift + | half_join(out, 0, (a, b, c) -> ((b, c), a)) + | filter(((b, c), _) -> (b, c) gt) + | half_join(pairs, 0, ((b, c), a, ()) -> (a, (b, c))); +-- A change to E(a, c): find E(a, b) strictly earlier, then check E(b, c). +let p2 = out | lift + | half_join(out, 1, (a, c, b) -> ((b, c), a)) + | filter(((b, c), _) -> (b, c) gt) + | half_join(pairs, 0, ((b, c), a, ()) -> (a, (b, c))); +-- A change to E(b, c): find E(a, b) and then E(a, c), both strictly earlier. +let p3 = down | map(((b, c), ()) -> (b, c)) | lift + | half_join(rev, 1, (b, c, a) -> ((a, c), b)) + | half_join(pairs, 1, ((a, c), b, ()) -> (a, (b, c))); + +-- Each found triangle carries the time of the change that found it; `delay` puts it there. +let triangles = (p1 + p2 + p3) | delay((_, (_, t)) -> t) | map((a, (bc, _)) -> (a, bc)); + +output "triangles" = triangles; +output "total" = triangles | map(_ -> ((), ())) | count; diff --git a/interactive/examples/recipes/triangles.out b/interactive/examples/recipes/triangles.out new file mode 100644 index 000000000..c10767d9c --- /dev/null +++ b/interactive/examples/recipes/triangles.out @@ -0,0 +1,24 @@ +_1 ok installed "tri" +_2 ok fed "tri" input 0 at t=0 +_3 ok fed "tri" input 0 at t=0 +_4 ok fed "tri" input 0 at t=0 +_5 ok fed "tri" input 0 at t=0 +_6 ok fed "tri" input 0 at t=0 +_7 ok fed "tri" input 0 at t=0 +_8 ok fed "tri" input 0 at t=0 +_9 ok sealed "tri" input 0 to 1 +_10 data tuple() val=4 +_10 ok as-of=0 +_11 data 3 val=tuple(2,1) +_11 data 4 val=tuple(2,1) +_11 data 4 val=tuple(3,1) +_11 data 4 val=tuple(3,2) +_11 ok as-of=0 +_12 ok fed "tri" input 0 at t=1 +_13 ok sealed "tri" input 0 to 2 +_14 data tuple() val=2 +_14 ok as-of=1 +_15 data 4 val=tuple(3,1) +_15 data 4 val=tuple(3,2) +_15 ok as-of=1 +_16 ok bye diff --git a/interactive/examples/recipes/triangles.txt b/interactive/examples/recipes/triangles.txt new file mode 100644 index 000000000..85f78d716 --- /dev/null +++ b/interactive/examples/recipes/triangles.txt @@ -0,0 +1,22 @@ +# Triangles as a delta join: see triangles.ddp. +# Run from interactive/: ddir_server < examples/recipes/triangles.txt +load tri from examples/recipes/triangles.ddp + +# Time 0: all six edges among 1, 2, 3, 4 (four triangles), and the edge 4-5 (none). +feed tri 0 1 val=2 time=0 +feed tri 0 1 val=3 time=0 +feed tri 0 1 val=4 time=0 +feed tri 0 2 val=3 time=0 +feed tri 0 2 val=4 time=0 +feed tri 0 3 val=4 time=0 +feed tri 0 4 val=5 time=0 +seal tri 0 1 +peek tri::total +peek tri::triangles + +# Time 1: removing 1-2 leaves the two triangles through 3-4. +feed tri 0 1 val=2 time=1 diff=-1 +seal tri 0 2 +peek tri::total +peek tri::triangles +exit diff --git a/interactive/examples/recipes/upsert.ddp b/interactive/examples/recipes/upsert.ddp new file mode 100644 index 000000000..9964a9bb4 --- /dev/null +++ b/interactive/examples/recipes/upsert.ddp @@ -0,0 +1,25 @@ +-- Upsert: a table keyed by `int`, where a put replaces the key's value and a delete removes it. +-- +-- Inputs are multisets, so feeding `5 val=2` after `5 val=1` leaves both rows. +-- Here the table is instead a root feedback variable: at each time it holds what `now` held +-- one time earlier, and `now` is that table with this time's writes applied. +-- The client never retracts anything; it only feeds the writes. + +let puts = input 0 : (int, int); -- (key, value): set key to value +let dels = input 1 : (int, ()); -- key: remove key + +-- A write counts at its own time only: `lift` pairs each row with its time t, and `delay` +-- moves a copy to t + 1, where subtracting it retracts the row. +let put_now = puts - (puts | lift | delay((_, (_, (t))) -> (t, 1) add) | map((k, (v, _)) -> (k, v))); +let del_now = dels - (dels | lift | delay((_, (_, (t))) -> (t, 1) add) | map((k, _) -> (k, ()))); + +-- Several puts to one key at one time: the least value wins. A put beats a delete at one time. +let written = put_now | min; +let touched = (written | map((k, _) -> (k, ()))) + del_now | distinct; + +var table : (int, int); +let kept = table - (table | join(touched, (k, v, ()) -> (k, v))); +let now = kept + written; +set table = now; + +output "table" = now; diff --git a/interactive/examples/recipes/upsert.out b/interactive/examples/recipes/upsert.out new file mode 100644 index 000000000..e5343b8af --- /dev/null +++ b/interactive/examples/recipes/upsert.out @@ -0,0 +1,33 @@ +_1 ok installed "upsert" +_2 ok fed "upsert" input 0 at t=0 +_3 ok fed "upsert" input 0 at t=0 +_4 ok sealed "upsert" input 0 to 1 +_5 ok sealed "upsert" input 1 to 1 +_6 data 1 val=10 +_6 data 2 val=20 +_6 ok as-of=0 +_7 ok fed "upsert" input 0 at t=1 +_8 ok fed "upsert" input 0 at t=1 +_9 ok sealed "upsert" input 0 to 2 +_10 ok sealed "upsert" input 1 to 2 +_11 data 1 val=11 +_11 data 2 val=20 +_11 data 3 val=30 +_11 ok as-of=1 +_12 ok fed "upsert" input 1 at t=2 +_13 ok fed "upsert" input 0 at t=2 +_14 ok fed "upsert" input 0 at t=2 +_15 ok sealed "upsert" input 0 to 3 +_16 ok sealed "upsert" input 1 to 3 +_17 data 1 val=11 +_17 data 3 val=31 +_17 ok as-of=2 +_18 ok sealed "upsert" input 0 to 4 +_19 ok sealed "upsert" input 1 to 4 +_20 data 1 val=11 +_20 data 3 val=31 +_20 ok as-of=3 +_21 data 1 val=10 +_21 data 2 val=20 +_21 ok as-of=0 +_22 ok bye diff --git a/interactive/examples/recipes/upsert.txt b/interactive/examples/recipes/upsert.txt new file mode 100644 index 000000000..6b06dfb60 --- /dev/null +++ b/interactive/examples/recipes/upsert.txt @@ -0,0 +1,34 @@ +# Upsert with a root feedback variable: see upsert.ddp. +# Run from interactive/: ddir_server < examples/recipes/upsert.txt +load upsert from examples/recipes/upsert.ddp + +# Time 0: put keys 1 and 2. +feed upsert 0 1 val=10 time=0 +feed upsert 0 2 val=20 time=0 +seal upsert 0 1 +seal upsert 1 1 +peek upsert::table + +# Time 1: overwrite key 1, and add key 3. +feed upsert 0 1 val=11 time=1 +feed upsert 0 3 val=30 time=1 +seal upsert 0 2 +seal upsert 1 2 +peek upsert::table + +# Time 2: delete key 2, and put key 3 twice: the least value wins. +feed upsert 1 2 time=2 +feed upsert 0 3 val=33 time=2 +feed upsert 0 3 val=31 time=2 +seal upsert 0 3 +seal upsert 1 3 +peek upsert::table + +# Time 3: no writes; the table stays as it was. +seal upsert 0 4 +seal upsert 1 4 +peek upsert::table + +# Reading back in time still works. +peek upsert::table as-of 0 +exit diff --git a/interactive/server/tests/recipes.rs b/interactive/server/tests/recipes.rs new file mode 100644 index 000000000..13ec852f9 --- /dev/null +++ b/interactive/server/tests/recipes.rs @@ -0,0 +1,66 @@ +//! The recipes in `examples/recipes`: each `NAME.txt` session, piped into a server's stdin from +//! `interactive/`, must print its recorded transcript `NAME.out`, at one worker and at three. +//! `UPDATE_RECIPES=1` records the one-worker transcripts instead. + +use std::io::Read; +use std::path::{Path, PathBuf}; +use std::process::{Command, Stdio}; +use std::time::{Duration, Instant}; + +fn interactive() -> PathBuf { Path::new(env!("CARGO_MANIFEST_DIR")).join("..") } + +/// The server's stdout for one session. An install's operator count is left out, so a change +/// to the optimizer does not change a recipe's transcript. +fn transcript(session: &Path, workers: usize) -> String { + let mut child = Command::new(env!("CARGO_BIN_EXE_ddir_server")) + .current_dir(interactive()) + .env("DDIR_WORKERS", workers.to_string()) + .env("DDIR_BIND", "127.0.0.1:0") + .env("DDIR_WS_BIND", "127.0.0.1:0") + .stdin(std::fs::File::open(session).unwrap()) + .stdout(Stdio::piped()) + .stderr(Stdio::null()) + .spawn() + .unwrap(); + let deadline = Instant::now() + Duration::from_secs(60); + while child.try_wait().unwrap().is_none() { + if Instant::now() > deadline { + let _ = child.kill(); + panic!("{}: the server did not exit", session.display()); + } + std::thread::sleep(Duration::from_millis(10)); + } + let mut out = String::new(); + child.stdout.take().unwrap().read_to_string(&mut out).unwrap(); + out.lines() + .map(|line| match line.find(" ok installed ") { + Some(_) => line.rsplit_once(" (").map_or(line, |(head, _)| head).to_string(), + None => line.to_string(), + }) + .collect::>() + .join("\n") + "\n" +} + +#[test] +fn recipes_print_their_recorded_transcripts() { + let dir = interactive().join("examples/recipes"); + let mut sessions: Vec = std::fs::read_dir(&dir).unwrap() + .map(|entry| entry.unwrap().path()) + .filter(|path| path.extension().is_some_and(|e| e == "txt")) + .collect(); + sessions.sort(); + assert!(!sessions.is_empty(), "no recipes in {}", dir.display()); + let update = std::env::var("UPDATE_RECIPES").as_deref() == Ok("1"); + for session in &sessions { + let recorded = session.with_extension("out"); + let got = transcript(session, 1); + if update { + std::fs::write(&recorded, &got).unwrap(); + } else { + let want = std::fs::read_to_string(&recorded) + .unwrap_or_else(|e| panic!("{}: {e}", recorded.display())); + assert_eq!(got, want, "{} disagrees with {}", session.display(), recorded.display()); + } + assert_eq!(transcript(session, 3), got, "{} at three workers", session.display()); + } +} From a58e7215b404f33903b0f73d08c907bf665070f7 Mon Sep 17 00:00:00 2001 From: Frank McSherry Date: Thu, 19 Nov 2026 22:44:01 -0500 Subject: [PATCH 2/6] DDIR: a draft language reference `LANGUAGE.md` gathers what was spread across the `pipe.rs` doc comment, corgi's op table and the README: collections and shapes, statements and sources, every collection operator, join fragments, lambdas and the scalar operations, time and scopes, iteration including root variables, and the language's design positions. Each idiom links to the recipe that shows it. The fragment doc comment in `pipe.rs` still said a one-variable fragment is keyed by `(x)`; since #1096 it is keyed by the variable itself. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5 --- interactive/LANGUAGE.md | 274 ++++++++++++++++++++++++++++++++++ interactive/README.md | 2 +- interactive/src/parse/pipe.rs | 4 +- 3 files changed, 277 insertions(+), 3 deletions(-) create mode 100644 interactive/LANGUAGE.md diff --git a/interactive/LANGUAGE.md b/interactive/LANGUAGE.md new file mode 100644 index 000000000..3897e59c9 --- /dev/null +++ b/interactive/LANGUAGE.md @@ -0,0 +1,274 @@ +# The DDIR language + +DDIR programs describe collections that change over time, and the server keeps every collection up to date as its inputs change. +This page is the reference: what a program is made of, every collection operator, the scalar language inside operators, time, iteration, and the positions the language takes on purpose. +For worked examples of the idioms that make programs fast or correct, see the [recipes](examples/recipes/README.md). +For the commands that load programs, feed them and read them, see the [server reference](server/README.md#server-command-reference). + +## A first program + +``` +-- Total sales per store, kept current as sales and refunds arrive. +let sales = input 0 : (int, int); -- (store, amount) + +output "total" = sales + | join_function((store, amount) -> ((store, ()), 0, amount) enlist) + | count; +``` + +A session loads it, feeds rows at times, closes those times, and reads the result: + +``` +load sum from examples/recipes/sum.ddp +feed sum 0 1 val=5 time=0 +feed sum 0 2 val=40 time=0 +seal sum 0 1 +peek sum::total +``` + +Later feeds at later times update `total`; nothing is recomputed from scratch. +This program is the [`sum`](examples/recipes/sum.ddp) recipe. + +## Collections + +A collection is a multiset of rows, each a `(key, value)` pair, that changes over time. +Each change is an update: a row, the time it happens, and a signed integer diff that adds to the row's count. +A row is present at a time when the sum of its diffs up to that time is not zero. + +Operators that group rows, such as `join`, `reduce` and `min`, group them by key. +When the whole row should be the key, use the unit value: `((a, b), ())` is a row keyed by the pair `(a, b)`. + +### Shapes + +Every collection has a shape, written `(key_shape, val_shape)`. + +| Shape | Meaning | +|---|---| +| `int` | a signed 64-bit integer | +| `float` | a 64-bit float | +| `()` | unit, the tuple of no fields | +| `(a, b, ...)` | a tuple | +| `List(a)` | a list; a string is `List(int)` of its UTF-8 bytes | +| `Option(a)`, `Result(a, b)` | the built-in sums | +| a declared name | an `enum` or `type` declared in the program | + +`(int, int)` is keyed by the first integer with the second as its value. +`((int, int), ())` is keyed by both, with a unit value. + +Parentheses always build a tuple: `(int)` is a tuple of one `int`, and `(x)` is a tuple of one value. +Write `int` and `x` where you mean the value itself; the two are different shapes, and mixing them is a shape error. + +### Declarations + +``` +enum Name = Known List(int) | Unknown; -- a sum type; a variant's payload is () when omitted +type Pair = (int, int); -- an alias, erased before planning +``` + +Types are global to the program, may appear in any order, and cannot be recursive. +Aliases are not yet available inside lambdas. + +## Statements + +| Statement | Meaning | +|---|---| +| `let x = expr;` | name a collection | +| `output "x" = expr;` | publish a collection as `PROGRAM::x`; root scope only, and each name once per scope | +| `name: { ... }` | a nested scope, which iterates to a fixed point (see [Iteration](#iteration)) | +| `var x;` or `var x : (k, v);` | declare a feedback variable, optionally with its shape | +| `set x = expr;` | bind a feedback variable | +| `enum ...;`, `type ...;` | declare types | + +A `--` starts a comment to the end of the line, inside lambdas too. +Within a scope the order of statements does not matter. + +### Sources + +| Source | Meaning | +|---|---| +| `input N : (k, v)` | input number `N`, with its shape | +| `input : (k, v)` | the next input number, in the order inputs are written | +| `input "name" : (k, v)` | the next input number, also fed or bound by `name` | +| `import "prog::x"` | another program's published output, optionally with `: (k, v)` | +| `name`, `scope::name` | a binding in this scope, an enclosing one, or a child scope's result | + +An input must declare its shape. +An import may declare one, which must match the shape the output is published at. + +## Operators + +An expression is a source followed by operator stages: `source | op | op`. +`+ b` and `- b` are stages too, and each stage applies to everything to its left: `a | map(f) + b | distinct` is `((a | map(f)) + b) | distinct`. +To add a whole pipeline, parenthesize it. + +Each operator's per-row logic is a lambda, written `(pattern -> body)`; see [Lambdas](#lambdas). + +### Row by row + +| Operator | Result | +|---|---| +| `map((k, v) -> (k2, v2))` | each row reshaped to the lambda's `(key, value)` | +| `value((k, v) -> v2)` | each row's value replaced, keeping its key | +| `filter((k, v) -> mask)` | the rows where the lambda gives a nonzero `int` | +| `flatmap((k, v) -> list)` | any number of rows per row: the lambda's `List` of `(key, value)` | +| `negate` | every diff negated | +| `a + b` | the union of updates, adding diffs | +| `a - b` | `a + (b \| negate)` | + +### Joins + +| Operator | Result | +|---|---| +| `join(other, (k, v1, v2) -> (k2, v3))` | for each key, every pair of a row on the left and a row on the right; diffs multiply | +| `half_join(other, strict, (k, v1, v2) -> (k2, v3))` | for a lifted stream `(k, (v, t))`: each update meets `other`'s rows whose times come before `t` (strictly when `strict` is 1); `other`'s own changes produce nothing | + +`half_join` is the building block of delta joins, which never build the intermediate results of a multiway join; see the [`triangles`](examples/recipes/triangles.ddp) recipe. + +#### Join fragments + +A join fragment writes a multiway join as a rule. +Each atom names a collection and its key and value patterns, and `*` joins the atoms left to right: + +``` +let revenue = orders(o, (c, _)) * customer(c, (_, n)) * nation(n, (name, r)) + | map(((o, c, n, name, r), ()) -> (name, o)); +``` + +A fragment's rows are keyed by its variables in order of first appearance, with a unit value; one variable is the key itself, and no variables are `()`. +A repeated variable tests equality, and a literal in a pattern tests that value. +Every later atom's key must use a variable bound by an earlier atom; the join order is the order written, so put selective atoms first. +Fragments are experimental: the parser expands them into plain joins, and they may change. + +### Reductions + +Each reduction groups rows by key and gives each key's result. + +| Operator | Result per key | +|---|---| +| `count` | the sum of the diffs of the key's rows | +| `min` | the least value | +| `distinct` | each present `(key, value)` row once | +| `collect` | a `List` of the values, each repeated by its count | +| `reduce((k, vals) -> out)` | the lambda's `out`, a `List` of `(value, diff)`, from `vals`, the key's `List` of `(value, diff)` in value order | +| `reduce(limit n, (k, vals) -> out)` | the same, with `vals` only the key's first `n` values | + +`count` adds diffs, which makes it the way to sum integers: move the value into the diff with `join_function`, then `count` ([`sum`](examples/recipes/sum.ddp)). +A key whose diffs sum to zero has no row, so a total of zero is absent. +There is no `max` or top-k operator; negate the value and use `min` or `reduce(limit n, ...)`, through a tree of groups for large inputs ([`top_k`](examples/recipes/top_k.ddp)). + +### Time + +| Operator | Result | +|---|---| +| `lift` | each value paired with its row's time: `(k, (v, t))` | +| `delay((k, v) -> t)` | each row moved to at least time `t` | +| `join_function((k, v) -> list)` | each row replaced by the lambda's `List` of `((key, value), time, diff)`, at the row's time joined with `time`, diffs multiplied | +| `lift_iter` | each value paired with the innermost loop's round | + +`delay` and `join_function` take times in the shape of the scope's time (see [Time](#time-and-scopes)); a coordinate of `0` leaves that coordinate as it is. +`delay` retracts rows after a fixed time ([`retention`](examples/recipes/retention.ddp)), orders work within a loop ([`connected_components`](examples/recipes/connected_components.ddp), [`shortest_paths`](examples/recipes/shortest_paths.ddp)), and with `lift` makes an input row count at its own time only ([`upsert`](examples/recipes/upsert.ddp)). + +### Arrangements and inspection + +| Operator | Result | +|---|---| +| `arrange` | the collection, indexed by key; joins and reductions arrange their inputs themselves, and share an arrangement when they can | +| `inspect(label)` | the collection unchanged, with its updates printed to the server's stdout | + +## Lambdas + +A lambda runs `let pattern = input in body`, where `input` is the tuple of the operator's rows: `(key, value)` for a row-by-row operator, `(key, left value, right value)` for a join, and `(key, vals)` for `reduce`. +The language is corgi's. + +Patterns are names, `_`, and tuples of patterns, each optionally with a shape: `((src, dst), ())`, `(k, v : int)`. +Operators follow their operands: `(a, b) add`, `xs len`, `(mask, a, b) select`. +Stages chain by juxtaposition, `xs map (x -> (x, 1) add) fold_add`, so nothing needs parentheses for grouping. + +| Form | Meaning | +|---|---| +| `x.0` | a tuple's field | +| `(a, b)`, `(a)`, `()` | a tuple, a one-field tuple, unit | +| `let p = e in body` | bind a value | +| `7`, `-3`, `2.5`, `"text"` | literals; a string is its `List(int)` of bytes | +| `[1, 2]`, `[] : List(int)` | a constant list | +| `xs map (x -> e)` | apply to each element | +| `(seed, xs) fold ((acc, x) -> e)`, `scan` | a fold, and its running form | +| `x inject Variant` | build a sum | +| `s match (A (x -> e1), B (y -> e2))` | take a sum apart | +| `s map_variant A (x -> e)` | change one variant | +| `(x, c) branch Option map_variant None (_ -> ())` | `Some(x)` when `c` is nonzero, else `None` | +| `x call name` | a host function, such as `fsqrt` or `bit_length` | +| `e : shape` | an ascription: `e` has this shape | + +A lambda reads only its own input and literals; to use a list beside another value inside `map`, pair them first with `cap_list`. + +### Scalar operations + +Integers are signed 64-bit; arithmetic takes its kind from its operands, so `(1, 2.5) add` is a shape error. + +| Group | Operations | +|---|---| +| arithmetic | `add sub mul div rem neg min max`; `to_float` | +| comparison | `eq ne lt le gt ge` give `int` 0 or 1, comparing any shapes in structural order, strings included | +| logic | `(a, b) and`, `or`, `xor` bitwise; `x and 255` masks; `(a, b, c) all`, `any` | +| choice | `(mask, then, else) select` | +| 64-bit words | `add_b64 sub_b64 mul_b64`; `shl_b64 k shr_b64 k rotl_b64 k rotr_b64 k` | +| lists | `len enlist append flatten range zip head get gather slices chunk k find cut adjacent filter sort dedup group sort_by sort_limit k transpose` | +| list reductions | `fold_add fold_mul fold_min fold_max fold_all fold_any`, and `scan_` of each | +| capture | `cap_list` pairs a context with every list element; `cap_sum` distributes one into every sum lane | +| checked forms | `try_get try_gather try_zip try_chunk` give a two-lane sum, whose second lane holds the rows the plain form would have lost something on | +| text | `split ","`, `parse_int` | +| other | `hash` (a stable 64-bit content hash), `unit` | +| host functions, by `call` | `fabs fsqrt fexp fln ffloor fceil fround fsin fcos ftan fpow fpowi fint fmin fmax`; `feq fne flt fle fgt fge` (IEEE comparisons); `bit_length` | + +`zip`, `get` and `gather` read the zero of the element's shape past the end of a list, and `chunk` drops a short last piece; the `try_` forms report those cases. + +## Time and scopes + +A program's root time is one integer by default, the time the client gives each feed. +A program loaded with `width=N` has root times of `N` coordinates, written `[x,y]` in commands, compared coordinate by coordinate. +The client advances an input's time with `seal`; a read at time `t` is complete once every input and import it depends on is past `t`. + +Inside a nested scope the time gains one coordinate per level, the loop's round: `(epoch, round)` one level down. +A `delay` or `join_function` there names every coordinate, as in `delay((_, d) -> (0, d))`. +`lift` gives the whole time, a tuple even at the root: `(k, (v, (t)))` for a one-coordinate time. + +## Iteration + +A nested scope, `name: { ... }`, computes a fixed point. + +1. Each feedback variable starts empty. +2. Each round, every variable becomes its `set` value, computed from the previous round's values. +3. The scope's result is the value each variable stops changing at. + +From outside, `name::x` is the fixed point of `x`. +A scope can hold negation, reductions and further scopes. +A recursion that never stops changing never finishes, and holds up every program on the server. + +Prioritized iteration uses `delay` to hold some rows back to later rounds, so that work likely to be undone waits ([`connected_components`](examples/recipes/connected_components.ddp)). +It can make the first computation much faster, and some later edits slower. + +At the root, `var` and `set` carry state from one time to the next instead. +A root variable is empty at first, and at each time `t` holds what its `set` held at `t - 1`. +Nothing iterates, and anything may feed the `set`, as the cycle passes through the step ([`upsert`](examples/recipes/upsert.ddp)). +Root variables need a one-coordinate root time. + +## Design positions + +These are deliberate, and differ from what SQL users expect. + +- **Inputs are multisets.** Feeding a row twice gives it a count of 2, and retracting an absent row gives it a count of -1. `distinct` keeps any row whose count is not zero. Replacing a key's value is a program's job ([`upsert`](examples/recipes/upsert.ddp)). +- **Arithmetic does not fail.** Integers wrap: `(9223372036854775807, 1) add` is `-9223372036854775808`. `(x, 0) div` is 0 and `(x, 0) rem` is `x`; division rounds toward zero, so `(-7, 2) div` is -3. A program that needs errors checks its operands and builds a `Result`. +- **There is no NULL.** A missing value is a variant. Declare an enum for it ([`left_join`](examples/recipes/left_join.ddp)); `() inject None` is refused, because `Option`'s `Some` lane has no shape until a value supplies one. +- **Nothing is durable.** A server's programs and data last as long as its process. +- **One server is one trust domain.** Programs share one command queue and one set of workers: a slow program delays the others, and a program that never finishes or crashes a worker stops them all. +- **Inputs belong to their program.** Another program cannot read an input, only an output. Changing a program's schema is a migration; a stable program that only publishes its inputs, read by programs that import them, keeps views replaceable. +- **There are no dates or times of day.** Use integers, such as days or seconds since an epoch. +- **Joins match keys exactly.** A range or approximate join is a re-keying into buckets first. +- **Strings are byte lists.** A feed and a peek spell `"ann"` as `list(97,110,110)`. + +## Seeing what a program does + +`ddir-plan file.ddp` prints a program's plan and its inferred shapes without running it, and exits non-zero when a program has a problem. +`peek` reads an output at a time, `tail` follows its changes, and `inspect(label)` prints a collection's updates to the server's stdout. +Only outputs can be read, so to see an intermediate collection, publish it as an output. diff --git a/interactive/README.md b/interactive/README.md index e1fd09581..a89614d80 100644 --- a/interactive/README.md +++ b/interactive/README.md @@ -1,6 +1,6 @@ # DDIR -[Server command reference](server/README.md#server-command-reference) · [Recipes](examples/recipes/README.md) +[Language reference](LANGUAGE.md) · [Recipes](examples/recipes/README.md) · [Server command reference](server/README.md#server-command-reference) An interpreted language (and runtime) for computation over continually changing collections. diff --git a/interactive/src/parse/pipe.rs b/interactive/src/parse/pipe.rs index da07da3af..7d43b3368 100644 --- a/interactive/src/parse/pipe.rs +++ b/interactive/src/parse/pipe.rs @@ -87,8 +87,8 @@ //! Reusing a variable tests equality. Plain filters run as soon as their fields are bound, //! before the next rekey; a direct first join precedes its first atom's tests. Variants bind //! through typed payload lists in `flatmap`, which can also emit the next lookup key. -//! Rows contain each variable once, in first-appearance order, keyed by their tuple (`(x)` for -//! one variable, `()` for none), with unit value. A test-free final layout composes into +//! Rows contain each variable once, in first-appearance order, keyed by their tuple (the +//! variable itself for one, `()` for none), with unit value. A test-free final layout composes into //! the last join's projection. Weights multiply; nothing implies distinct. //! Even the parsed view shows the expansion. //! From a4ed8db185e2400bb84e11f699806427edb0c0a6 Mon Sep 17 00:00:00 2001 From: Frank McSherry Date: Fri, 20 Nov 2026 05:51:47 -0500 Subject: [PATCH 3/6] DDIR reference: operators by reference form and sugar Each operator group names its reference form, and each sugar shows its expansion. join_function is the reference for linear operators, join for joins, and reduce for reductions. lift, lift_iter and half_join move to their own group, as they read times and are not functions of data alone. Every expansion was run against its sugar on the server, with equal results. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5 --- interactive/LANGUAGE.md | 75 ++++++++++++++++++++++++++--------------- 1 file changed, 47 insertions(+), 28 deletions(-) diff --git a/interactive/LANGUAGE.md b/interactive/LANGUAGE.md index 3897e59c9..085e381c9 100644 --- a/interactive/LANGUAGE.md +++ b/interactive/LANGUAGE.md @@ -103,26 +103,40 @@ To add a whole pipeline, parenthesize it. Each operator's per-row logic is a lambda, written `(pattern -> body)`; see [Lambdas](#lambdas). -### Row by row +Each group of operators below has one reference form, and the other operators in the group are sugar for it. +A sugar's expansion is shown next to it, and writing the expansion gives the same result. -| Operator | Result | +### Linear operators + +The reference form is `join_function((k, v) -> list)`. +Each row becomes the lambda's `List` of `((key, value), time, diff)`, at the row's time joined with `time`, with the two diffs multiplied. +The `time` is in the shape of the scope's time (see [Time](#time-and-scopes)), and a coordinate of `0` leaves that coordinate as it is. + +| Sugar | Expansion | |---|---| -| `map((k, v) -> (k2, v2))` | each row reshaped to the lambda's `(key, value)` | -| `value((k, v) -> v2)` | each row's value replaced, keeping its key | -| `filter((k, v) -> mask)` | the rows where the lambda gives a nonzero `int` | -| `flatmap((k, v) -> list)` | any number of rows per row: the lambda's `List` of `(key, value)` | -| `negate` | every diff negated | -| `a + b` | the union of updates, adding diffs | -| `a - b` | `a + (b \| negate)` | +| `map((k, v) -> (k2, v2))` | `join_function((k, v) -> ((k2, v2), 0, 1) enlist)` | +| `value((k, v) -> v2)` | `map((k, v) -> (k, v2))` | +| `filter((k, v) -> mask)` | `join_function((k, v) -> (mask, ((k, v), 0, 1)) enlist filter)` | +| `flatmap((k, v) -> rows)` | `join_function((k, v) -> rows map (r -> (r, 0, 1)))` | +| `negate` | `join_function((k, v) -> ((k, v), 0, -1) enlist)` | +| `delay((k, v) -> t)` | `join_function((k, v) -> ((k, v), t, 1) enlist)` | + +A `filter` keeps the rows where `mask` is a nonzero `int`. +Inside a nested scope each `0` above is a time of every coordinate, such as `(0, 0)` one level down. + +`delay` retracts rows after a fixed time ([`retention`](examples/recipes/retention.ddp)) and orders work within a loop ([`connected_components`](examples/recipes/connected_components.ddp), [`shortest_paths`](examples/recipes/shortest_paths.ddp)). + +`a + b` is the union of two collections, adding diffs. +It is its own operator, as it reads two inputs, and `a - b` is sugar for `a + (b | negate)`. + +The operators in this group are functions of data: a result at time `t` depends only on the input's rows at `t`, not on when they arrived. ### Joins -| Operator | Result | -|---|---| -| `join(other, (k, v1, v2) -> (k2, v3))` | for each key, every pair of a row on the left and a row on the right; diffs multiply | -| `half_join(other, strict, (k, v1, v2) -> (k2, v3))` | for a lifted stream `(k, (v, t))`: each update meets `other`'s rows whose times come before `t` (strictly when `strict` is 1); `other`'s own changes produce nothing | +The reference form is `join(other, (k, v1, v2) -> (k2, v3))`. +For each key it pairs every row on the left with every row on the right, and multiplies their diffs. -`half_join` is the building block of delta joins, which never build the intermediate results of a multiway join; see the [`triangles`](examples/recipes/triangles.ddp) recipe. +[Join fragments](#join-fragments) are sugar for a chain of `join`s. #### Join fragments @@ -141,32 +155,37 @@ Fragments are experimental: the parser expands them into plain joins, and they m ### Reductions -Each reduction groups rows by key and gives each key's result. +The reference form is `reduce((k, vals) -> out)`, or `reduce(limit n, (k, vals) -> out)`. +It groups rows by key. +`vals` is the key's `List` of `(value, diff)` in value order, the values whose diffs do not sum to zero, and only the first `n` of them with a limit. +`out` is a `List` of `(value, diff)`, the key's result, and a key whose `out` is empty has no row. -| Operator | Result per key | -|---|---| -| `count` | the sum of the diffs of the key's rows | -| `min` | the least value | -| `distinct` | each present `(key, value)` row once | -| `collect` | a `List` of the values, each repeated by its count | -| `reduce((k, vals) -> out)` | the lambda's `out`, a `List` of `(value, diff)`, from `vals`, the key's `List` of `(value, diff)` in value order | -| `reduce(limit n, (k, vals) -> out)` | the same, with `vals` only the key's first `n` values | +| Sugar | Result per key | Expansion | +|---|---|---| +| `min` | the least value | `reduce(limit 1, (k, vals) -> (vals head .0, 1) enlist)` | +| `count` | the sum of the diffs | `reduce((k, vals) -> let n = vals map ((_, d) -> d) fold_add in (n, (n, 1)) enlist filter)` | +| `distinct` | each present value once | `reduce((k, vals) -> vals map ((v, _) -> (v, 1)))` | +| `collect` | a `List` of the values, each repeated by its count | `reduce((k, vals) -> (vals map ((v, d) -> (v, (0, d) range) cap_list map ((v, _) -> v)) flatten .1, 1) enlist)` | + +`distinct` is lowered by whole record rather than by key when that is cheaper; the result is the same. `count` adds diffs, which makes it the way to sum integers: move the value into the diff with `join_function`, then `count` ([`sum`](examples/recipes/sum.ddp)). A key whose diffs sum to zero has no row, so a total of zero is absent. There is no `max` or top-k operator; negate the value and use `min` or `reduce(limit n, ...)`, through a tree of groups for large inputs ([`top_k`](examples/recipes/top_k.ddp)). -### Time +### Operators that read time + +These operators are not functions of data alone: their results depend on the times at which rows arrived, and not only on the rows present at each time. +None of them is sugar. | Operator | Result | |---|---| | `lift` | each value paired with its row's time: `(k, (v, t))` | -| `delay((k, v) -> t)` | each row moved to at least time `t` | -| `join_function((k, v) -> list)` | each row replaced by the lambda's `List` of `((key, value), time, diff)`, at the row's time joined with `time`, diffs multiplied | | `lift_iter` | each value paired with the innermost loop's round | +| `half_join(other, strict, (k, v1, v2) -> (k2, v3))` | for a lifted stream `(k, (v, t))`: each update meets `other`'s rows whose times come before `t` (strictly when `strict` is 1); `other`'s own changes produce nothing | -`delay` and `join_function` take times in the shape of the scope's time (see [Time](#time-and-scopes)); a coordinate of `0` leaves that coordinate as it is. -`delay` retracts rows after a fixed time ([`retention`](examples/recipes/retention.ddp)), orders work within a loop ([`connected_components`](examples/recipes/connected_components.ddp), [`shortest_paths`](examples/recipes/shortest_paths.ddp)), and with `lift` makes an input row count at its own time only ([`upsert`](examples/recipes/upsert.ddp)). +`half_join` is the building block of delta joins, which never build the intermediate results of a multiway join; see the [`triangles`](examples/recipes/triangles.ddp) recipe. +`lift` with `delay` makes an input row count at its own time only ([`upsert`](examples/recipes/upsert.ddp)). ### Arrangements and inspection From e697dc47e494eecf0d0eaf0062f6697a5d235529 Mon Sep 17 00:00:00 2001 From: Frank McSherry Date: Fri, 20 Nov 2026 06:24:24 -0500 Subject: [PATCH 4/6] DDIR reference: count drops values, no value or lift_iter This follows the ddir-prune branch, which removes value and lift_iter and has count project its values away. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5 --- interactive/LANGUAGE.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/interactive/LANGUAGE.md b/interactive/LANGUAGE.md index 085e381c9..d8dacd2b7 100644 --- a/interactive/LANGUAGE.md +++ b/interactive/LANGUAGE.md @@ -115,7 +115,6 @@ The `time` is in the shape of the scope's time (see [Time](#time-and-scopes)), a | Sugar | Expansion | |---|---| | `map((k, v) -> (k2, v2))` | `join_function((k, v) -> ((k2, v2), 0, 1) enlist)` | -| `value((k, v) -> v2)` | `map((k, v) -> (k, v2))` | | `filter((k, v) -> mask)` | `join_function((k, v) -> (mask, ((k, v), 0, 1)) enlist filter)` | | `flatmap((k, v) -> rows)` | `join_function((k, v) -> rows map (r -> (r, 0, 1)))` | | `negate` | `join_function((k, v) -> ((k, v), 0, -1) enlist)` | @@ -163,11 +162,11 @@ It groups rows by key. | Sugar | Result per key | Expansion | |---|---|---| | `min` | the least value | `reduce(limit 1, (k, vals) -> (vals head .0, 1) enlist)` | -| `count` | the sum of the diffs | `reduce((k, vals) -> let n = vals map ((_, d) -> d) fold_add in (n, (n, 1)) enlist filter)` | +| `count` | the sum of the diffs | `map((k, _) -> (k, ())) \| reduce((k, vals) -> (vals head .1, 1) enlist)` | | `distinct` | each present value once | `reduce((k, vals) -> vals map ((v, _) -> (v, 1)))` | | `collect` | a `List` of the values, each repeated by its count | `reduce((k, vals) -> (vals map ((v, d) -> (v, (0, d) range) cap_list map ((v, _) -> v)) flatten .1, 1) enlist)` | -`distinct` is lowered by whole record rather than by key when that is cheaper; the result is the same. +When the value is not `()`, `count` drops it first and `distinct` keys by the whole record, so neither reduce reads a key's values together. `count` adds diffs, which makes it the way to sum integers: move the value into the diff with `join_function`, then `count` ([`sum`](examples/recipes/sum.ddp)). A key whose diffs sum to zero has no row, so a total of zero is absent. @@ -181,10 +180,10 @@ None of them is sugar. | Operator | Result | |---|---| | `lift` | each value paired with its row's time: `(k, (v, t))` | -| `lift_iter` | each value paired with the innermost loop's round | -| `half_join(other, strict, (k, v1, v2) -> (k2, v3))` | for a lifted stream `(k, (v, t))`: each update meets `other`'s rows whose times come before `t` (strictly when `strict` is 1); `other`'s own changes produce nothing | +| `half_join(other, strict, (k, v1, v2) -> (k2, v3))` | for a lifted stream `(k, (v, t))`: each update meets `other`'s rows whose times come before `t` (strictly when `strict`, a bool written `0` or `1`, is `1`); `other`'s own changes produce nothing | `half_join` is the building block of delta joins, which never build the intermediate results of a multiway join; see the [`triangles`](examples/recipes/triangles.ddp) recipe. +A loop's round is the last coordinate of `lift`'s time. `lift` with `delay` makes an input row count at its own time only ([`upsert`](examples/recipes/upsert.ddp)). ### Arrangements and inspection From 581c8c8fc5463d50c7435ecb77e0ce0486b71cb7 Mon Sep 17 00:00:00 2001 From: Frank McSherry Date: Fri, 20 Nov 2026 20:19:38 -0500 Subject: [PATCH 5/6] DDIR reference: inputs by label, bound at load; no import Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5 --- interactive/LANGUAGE.md | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/interactive/LANGUAGE.md b/interactive/LANGUAGE.md index d8dacd2b7..27d7c6bb5 100644 --- a/interactive/LANGUAGE.md +++ b/interactive/LANGUAGE.md @@ -86,14 +86,17 @@ Within a scope the order of statements does not matter. | Source | Meaning | |---|---| -| `input N : (k, v)` | input number `N`, with its shape | -| `input : (k, v)` | the next input number, in the order inputs are written | -| `input "name" : (k, v)` | the next input number, also fed or bound by `name` | -| `import "prog::x"` | another program's published output, optionally with `: (k, v)` | +| `input N : (k, v)` | the input labelled `N`, with its shape | +| `input "name" : (k, v)` | the input labelled `name`, an identifier, with its shape | | `name`, `scope::name` | a binding in this scope, an enclosing one, or a child scope's result | +An input's label is its identity, and the order inputs are written in never matters. +Writing a label again reads the same input. An input must declare its shape. -An import may declare one, which must match the shape the output is published at. + +A program does not name the traces it reads. +The client feeds an input by its label, or the load binds it to another program's output instead, as `load p from p.ddp edges=graph::edges`. +A bound input's shape must match the shape the output is published at. ## Operators @@ -245,7 +248,7 @@ Integers are signed 64-bit; arithmetic takes its kind from its operands, so `(1, A program's root time is one integer by default, the time the client gives each feed. A program loaded with `width=N` has root times of `N` coordinates, written `[x,y]` in commands, compared coordinate by coordinate. -The client advances an input's time with `seal`; a read at time `t` is complete once every input and import it depends on is past `t`. +The client advances an input's time with `seal`; a read at time `t` is complete once every input it depends on, fed or bound, is past `t`. Inside a nested scope the time gains one coordinate per level, the loop's round: `(epoch, round)` one level down. A `delay` or `join_function` there names every coordinate, as in `delay((_, d) -> (0, d))`. @@ -280,7 +283,7 @@ These are deliberate, and differ from what SQL users expect. - **There is no NULL.** A missing value is a variant. Declare an enum for it ([`left_join`](examples/recipes/left_join.ddp)); `() inject None` is refused, because `Option`'s `Some` lane has no shape until a value supplies one. - **Nothing is durable.** A server's programs and data last as long as its process. - **One server is one trust domain.** Programs share one command queue and one set of workers: a slow program delays the others, and a program that never finishes or crashes a worker stops them all. -- **Inputs belong to their program.** Another program cannot read an input, only an output. Changing a program's schema is a migration; a stable program that only publishes its inputs, read by programs that import them, keeps views replaceable. +- **Inputs belong to their program.** Another program cannot read an input, only an output. Changing a program's schema is a migration; a stable program that only publishes its inputs, read by programs whose inputs are bound to them, keeps views replaceable. - **There are no dates or times of day.** Use integers, such as days or seconds since an epoch. - **Joins match keys exactly.** A range or approximate join is a re-keying into buckets first. - **Strings are byte lists.** A feed and a peek spell `"ann"` as `list(97,110,110)`. From e53f80132f15931d1def76823911805f075b8896 Mon Sep 17 00:00:00 2001 From: Frank McSherry Date: Fri, 20 Nov 2026 20:40:53 -0500 Subject: [PATCH 6/6] DDIR recipes and reference: joins pass the joined row; use sum Since #1103 and #1104 a join's projection takes `(k, (l, r))` and a half join's takes `(k, ((l, r), t))`. The recipes are rewritten to those forms, and their transcripts are unchanged. The reference gives `join(other)` and `half_join(other, strict)` as the reference forms, with a projection as sugar for a `map`. The `sum` recipe and the first program use `| sum`, and the reductions table lists it with its expansion. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5 --- interactive/LANGUAGE.md | 20 +++++++++++-------- interactive/examples/recipes/README.md | 2 +- .../examples/recipes/connected_components.ddp | 2 +- interactive/examples/recipes/left_join.ddp | 4 ++-- .../examples/recipes/shortest_paths.ddp | 2 +- interactive/examples/recipes/sum.ddp | 10 +++------- interactive/examples/recipes/sum.txt | 2 +- interactive/examples/recipes/triangles.ddp | 12 +++++------ interactive/examples/recipes/upsert.ddp | 2 +- 9 files changed, 28 insertions(+), 28 deletions(-) diff --git a/interactive/LANGUAGE.md b/interactive/LANGUAGE.md index 27d7c6bb5..cf7e25abb 100644 --- a/interactive/LANGUAGE.md +++ b/interactive/LANGUAGE.md @@ -11,9 +11,7 @@ For the commands that load programs, feed them and read them, see the [server re -- Total sales per store, kept current as sales and refunds arrive. let sales = input 0 : (int, int); -- (store, amount) -output "total" = sales - | join_function((store, amount) -> ((store, ()), 0, amount) enlist) - | count; +output "total" = sales | sum; ``` A session loads it, feeds rows at times, closes those times, and reads the result: @@ -135,8 +133,12 @@ The operators in this group are functions of data: a result at time `t` depends ### Joins -The reference form is `join(other, (k, v1, v2) -> (k2, v3))`. -For each key it pairs every row on the left with every row on the right, and multiplies their diffs. +The reference form is `join(other)`. +For each key it pairs every row `(k, v1)` on the left with every row `(k, v2)` on the right, giving `(k, (v1, v2))`, and multiplies their diffs. + +| Sugar | Expansion | +|---|---| +| `join(other, (k, (v1, v2)) -> (k2, v3))` | `join(other) \| map((k, (v1, v2)) -> (k2, v3))`, without building the joined rows | [Join fragments](#join-fragments) are sugar for a chain of `join`s. @@ -166,12 +168,13 @@ It groups rows by key. |---|---|---| | `min` | the least value | `reduce(limit 1, (k, vals) -> (vals head .0, 1) enlist)` | | `count` | the sum of the diffs | `map((k, _) -> (k, ())) \| reduce((k, vals) -> (vals head .1, 1) enlist)` | +| `sum` | the sum of the integer values, each weighted by its diff | `lift \| join_function((k, (v, t)) -> ((k, ()), t, v) enlist) \| count` | | `distinct` | each present value once | `reduce((k, vals) -> vals map ((v, _) -> (v, 1)))` | | `collect` | a `List` of the values, each repeated by its count | `reduce((k, vals) -> (vals map ((v, d) -> (v, (0, d) range) cap_list map ((v, _) -> v)) flatten .1, 1) enlist)` | When the value is not `()`, `count` drops it first and `distinct` keys by the whole record, so neither reduce reads a key's values together. -`count` adds diffs, which makes it the way to sum integers: move the value into the diff with `join_function`, then `count` ([`sum`](examples/recipes/sum.ddp)). +`sum` moves each value into its diff and counts, so it costs what `count` costs. Its `lift` only hands `join_function` each row's own time, which leaves the time as it is at any depth ([`sum`](examples/recipes/sum.ddp)). A key whose diffs sum to zero has no row, so a total of zero is absent. There is no `max` or top-k operator; negate the value and use `min` or `reduce(limit n, ...)`, through a tree of groups for large inputs ([`top_k`](examples/recipes/top_k.ddp)). @@ -183,7 +186,8 @@ None of them is sugar. | Operator | Result | |---|---| | `lift` | each value paired with its row's time: `(k, (v, t))` | -| `half_join(other, strict, (k, v1, v2) -> (k2, v3))` | for a lifted stream `(k, (v, t))`: each update meets `other`'s rows whose times come before `t` (strictly when `strict`, a bool written `0` or `1`, is `1`); `other`'s own changes produce nothing | +| `half_join(other, strict)` | for a lifted stream `(k, (v1, t))`: each update meets `other`'s rows `(k, v2)` whose times come before its own (strictly when `strict`, a bool written `0` or `1`, is `1`), giving `(k, ((v1, v2), t2))` at the update's time, where `t2` is the later of the two times, coordinate by coordinate; `other`'s own changes produce nothing | +| `half_join(other, strict, f)` | `half_join(other, strict) \| map(f)`, without building the joined rows | `half_join` is the building block of delta joins, which never build the intermediate results of a multiway join; see the [`triangles`](examples/recipes/triangles.ddp) recipe. A loop's round is the last coordinate of `lift`'s time. @@ -198,7 +202,7 @@ A loop's round is the last coordinate of `lift`'s time. ## Lambdas -A lambda runs `let pattern = input in body`, where `input` is the tuple of the operator's rows: `(key, value)` for a row-by-row operator, `(key, left value, right value)` for a join, and `(key, vals)` for `reduce`. +A lambda runs `let pattern = input in body`, where `input` is the tuple of the operator's rows: `(key, value)` for a row-by-row operator, `(key, (left value, right value))` for a join, and `(key, vals)` for `reduce`. The language is corgi's. Patterns are names, `_`, and tuples of patterns, each optionally with a shape: `((src, dst), ())`, `(k, v : int)`. diff --git a/interactive/examples/recipes/README.md b/interactive/examples/recipes/README.md index 9947fcd2a..eadb765a1 100644 --- a/interactive/examples/recipes/README.md +++ b/interactive/examples/recipes/README.md @@ -16,7 +16,7 @@ After an intended change, `UPDATE_RECIPES=1 cargo test --release -p ddir-server | Recipe | Idiom | Use it for | |---|---|---| -| [`sum`](sum.ddp) | `join_function` moves a value into the diff, and `count` adds up the diffs | sums of integers, incrementally | +| [`sum`](sum.ddp) | `sum` adds each key's values, weighted by their diffs | sums of integers, incrementally | | [`top_k`](top_k.ddp) | a tree of `reduce(limit k)` groups | top k over many rows, with small updates | | [`retention`](retention.ddp) | `lift` and `delay` retract each row a fixed time after it arrives | sliding windows, expiry | | [`upsert`](upsert.ddp) | a root `var` holds the table one time ago; writes count at their own time only | a keyed table with put and delete | diff --git a/interactive/examples/recipes/connected_components.ddp b/interactive/examples/recipes/connected_components.ddp index 879046e64..5f99425c6 100644 --- a/interactive/examples/recipes/connected_components.ddp +++ b/interactive/examples/recipes/connected_components.ddp @@ -15,7 +15,7 @@ let nodes = edges | map((a, _) -> (a, ())) | distinct; cc: { -- The inner time is (epoch, round): `0` leaves the epoch alone and sets the round. let seeds = nodes | map((n, _) -> (n, n)) | delay((_, n) -> (0, (256, n call bit_length) mul)); - let proposals = labels | join(edges, (_, label, dst) -> (dst, label)); + let proposals = labels | join(edges, (_, (label, dst)) -> (dst, label)); set labels = seeds + proposals | min; } diff --git a/interactive/examples/recipes/left_join.ddp b/interactive/examples/recipes/left_join.ddp index 9e0ae2643..92b1e628a 100644 --- a/interactive/examples/recipes/left_join.ddp +++ b/interactive/examples/recipes/left_join.ddp @@ -11,7 +11,7 @@ let orders = input 0 : (int, int); -- (order, customer) let customers = input 1 : (int, List(int)); -- (customer, name); a string is a list of bytes let by_customer = orders | map((order, customer) -> (customer, order)); -let matched = by_customer | join(customers, (c, order, name) -> (order, (c, name inject Known))); -let unmatched = by_customer - (by_customer | join(customers, (c, order, _) -> (c, order))); +let matched = by_customer | join(customers, (c, (order, name)) -> (order, (c, name inject Known))); +let unmatched = by_customer - (by_customer | join(customers, (c, (order, _)) -> (c, order))); output "orders" = matched + (unmatched | map((c, order) -> (order, (c, () inject Unknown)))); diff --git a/interactive/examples/recipes/shortest_paths.ddp b/interactive/examples/recipes/shortest_paths.ddp index 8d20d543c..c2510fc82 100644 --- a/interactive/examples/recipes/shortest_paths.ddp +++ b/interactive/examples/recipes/shortest_paths.ddp @@ -14,7 +14,7 @@ let roots = input 1 : (int, ()); -- the sources, at distance 0 sssp: { let proposals = dist - | join(edges, (_, d, (dst, w)) -> (dst, (d, w) add)) + | join(edges, (_, (d, (dst, w))) -> (dst, (d, w) add)) | delay((_, d) -> (0, ((d, 64) div, 16) mul)); set dist = (roots | map((n, ()) -> (n, 0))) + proposals | min; } diff --git a/interactive/examples/recipes/sum.ddp b/interactive/examples/recipes/sum.ddp index bd94838a5..0d1969ca0 100644 --- a/interactive/examples/recipes/sum.ddp +++ b/interactive/examples/recipes/sum.ddp @@ -1,13 +1,9 @@ --- Sum an integer per key: move the value into the diff, and `count` adds up the diffs. +-- Sum an integer per key with `sum`, which adds each key's values weighted by their diffs. -- --- `join_function` replaces each row with a list of `((key, val), time, diff)`. --- Here each sale becomes one row `(store, ())` at its own time (`0` leaves the time as it is), --- with the amount as its diff; `count` then gives each store the total of its diffs. +-- `sum` is sugar: it moves each value into its row's diff, and `count` adds up the diffs. -- The sum is maintained incrementally, where folding a list of values would rebuild it per change. -- A total of zero has no row, like any record whose diffs cancel; it is bounded by `i64`. let sales = input 0 : (int, int); -- (store, amount); a refund is a negative amount -output "total" = sales - | join_function((store, amount) -> ((store, ()), 0, amount) enlist) - | count; +output "total" = sales | sum; diff --git a/interactive/examples/recipes/sum.txt b/interactive/examples/recipes/sum.txt index 4d58649a2..c93ddd3f6 100644 --- a/interactive/examples/recipes/sum.txt +++ b/interactive/examples/recipes/sum.txt @@ -1,4 +1,4 @@ -# Sum per key with `count`: see sum.ddp. +# Sum per key with `sum`: see sum.ddp. # Run from interactive/: ddir_server < examples/recipes/sum.txt load sum from examples/recipes/sum.ddp diff --git a/interactive/examples/recipes/triangles.ddp b/interactive/examples/recipes/triangles.ddp index 825943d38..da1f09204 100644 --- a/interactive/examples/recipes/triangles.ddp +++ b/interactive/examples/recipes/triangles.ddp @@ -20,18 +20,18 @@ let pairs = down | arrange; -- keyed by the whole edge -- A change to E(a, b): find E(a, c), then check E(b, c). let p1 = out | lift - | half_join(out, 0, (a, b, c) -> ((b, c), a)) + | half_join(out, 0, (a, ((b, c), t)) -> ((b, c), (a, t))) | filter(((b, c), _) -> (b, c) gt) - | half_join(pairs, 0, ((b, c), a, ()) -> (a, (b, c))); + | half_join(pairs, 0, ((b, c), ((a, ()), t)) -> (a, ((b, c), t))); -- A change to E(a, c): find E(a, b) strictly earlier, then check E(b, c). let p2 = out | lift - | half_join(out, 1, (a, c, b) -> ((b, c), a)) + | half_join(out, 1, (a, ((c, b), t)) -> ((b, c), (a, t))) | filter(((b, c), _) -> (b, c) gt) - | half_join(pairs, 0, ((b, c), a, ()) -> (a, (b, c))); + | half_join(pairs, 0, ((b, c), ((a, ()), t)) -> (a, ((b, c), t))); -- A change to E(b, c): find E(a, b) and then E(a, c), both strictly earlier. let p3 = down | map(((b, c), ()) -> (b, c)) | lift - | half_join(rev, 1, (b, c, a) -> ((a, c), b)) - | half_join(pairs, 1, ((a, c), b, ()) -> (a, (b, c))); + | half_join(rev, 1, (b, ((c, a), t)) -> ((a, c), (b, t))) + | half_join(pairs, 1, ((a, c), ((b, ()), t)) -> (a, ((b, c), t))); -- Each found triangle carries the time of the change that found it; `delay` puts it there. let triangles = (p1 + p2 + p3) | delay((_, (_, t)) -> t) | map((a, (bc, _)) -> (a, bc)); diff --git a/interactive/examples/recipes/upsert.ddp b/interactive/examples/recipes/upsert.ddp index 9964a9bb4..f400974cc 100644 --- a/interactive/examples/recipes/upsert.ddp +++ b/interactive/examples/recipes/upsert.ddp @@ -18,7 +18,7 @@ let written = put_now | min; let touched = (written | map((k, _) -> (k, ()))) + del_now | distinct; var table : (int, int); -let kept = table - (table | join(touched, (k, v, ()) -> (k, v))); +let kept = table - (table | join(touched, (k, (v, ())) -> (k, v))); let now = kept + written; set table = now;