From e3f36e8a5b0802c387173742fd3f56651e96a25b Mon Sep 17 00:00:00 2001 From: Frank McSherry Date: Sat, 21 Nov 2026 09:59:05 -0500 Subject: [PATCH] DDIR: examples cut to one session and six idioms examples/ now holds the access-control session the README walks through, and six programs that each show one idiom in their comments. Its README is a short index. The recipes' sessions, transcripts and test go, as do the server sessions and the sum and left_join recipes; LANGUAGE.md already covers both. The batch programs from examples/programs move to tests/programs, where the corgi backend test already read its other fixtures. The plan pairs no longer cover the examples, and the multiworker trim test writes its two programs inline. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01H6wB4Gjwwivtq59LaFYbF5 --- interactive/LANGUAGE.md | 32 ++--- interactive/README.md | 28 ++-- interactive/examples/README.md | 13 ++ .../examples/{server/programs => }/access.ddp | 0 .../examples/{server/sessions => }/access.txt | 6 +- interactive/examples/components.ddp | 19 +++ interactive/examples/recipes/README.md | 30 ---- .../examples/recipes/connected_components.ddp | 22 --- .../examples/recipes/connected_components.out | 27 ---- .../examples/recipes/connected_components.txt | 21 --- interactive/examples/recipes/left_join.ddp | 17 --- interactive/examples/recipes/left_join.out | 16 --- interactive/examples/recipes/left_join.txt | 19 --- interactive/examples/recipes/retention.ddp | 13 -- interactive/examples/recipes/retention.out | 16 --- interactive/examples/recipes/retention.txt | 17 --- .../examples/recipes/shortest_paths.ddp | 22 --- .../examples/recipes/shortest_paths.out | 23 ---- .../examples/recipes/shortest_paths.txt | 21 --- interactive/examples/recipes/sum.ddp | 9 -- interactive/examples/recipes/sum.out | 15 -- interactive/examples/recipes/sum.txt | 18 --- interactive/examples/recipes/top_k.ddp | 19 --- interactive/examples/recipes/top_k.out | 21 --- interactive/examples/recipes/top_k.txt | 22 --- interactive/examples/recipes/triangles.out | 24 ---- interactive/examples/recipes/triangles.txt | 22 --- interactive/examples/recipes/upsert.ddp | 25 ---- interactive/examples/recipes/upsert.out | 33 ----- interactive/examples/recipes/upsert.txt | 34 ----- interactive/examples/retention.ddp | 10 ++ interactive/examples/server/README.md | 130 ------------------ .../examples/server/programs/arrange_idem.ddp | 54 -------- interactive/examples/server/programs/echo.ddp | 5 - .../examples/server/programs/producer.ddp | 5 - .../examples/server/programs/reach_import.ddp | 15 -- .../examples/server/sessions/arrange_idem.txt | 39 ------ interactive/examples/server/sessions/drop.txt | 25 ---- interactive/examples/server/sessions/peek.txt | 19 --- .../examples/server/sessions/shared_trace.txt | 29 ---- .../examples/server/sessions/values.txt | 19 --- interactive/examples/shortest_paths.ddp | 18 +++ interactive/examples/top_k.ddp | 16 +++ .../examples/{recipes => }/triangles.ddp | 17 +-- interactive/examples/upsert.ddp | 21 +++ interactive/server/README.md | 2 +- interactive/server/tests/multiworker.rs | 18 ++- interactive/server/tests/recipes.rs | 66 --------- interactive/src/explain/mod.rs | 2 +- interactive/src/lower.rs | 2 +- interactive/tests/corgi_backend.rs | 13 +- interactive/tests/explain.rs | 2 +- interactive/tests/plan_pairs.rs | 2 +- interactive/tests/plan_pairs/README.md | 6 +- .../plan_pairs/expected/server_access.ddp | 41 ------ .../server_access.lowered--optimized.diff | 50 ------- .../server_access.optimized--projected.diff | 0 .../server_access.parsed--lowered.diff | 55 -------- .../expected/server_access.parsed.ddp | 19 --- .../server_access.projected--as-of.diff | 0 .../expected/server_arrange_idem.ddp | 44 ------ ...erver_arrange_idem.lowered--optimized.diff | 74 ---------- ...ver_arrange_idem.optimized--projected.diff | 0 .../server_arrange_idem.parsed--lowered.diff | 73 ---------- .../expected/server_arrange_idem.parsed.ddp | 18 --- .../server_arrange_idem.projected--as-of.diff | 0 .../tests/plan_pairs/expected/server_echo.ddp | 6 - .../server_echo.lowered--optimized.diff | 0 .../server_echo.optimized--projected.diff | 0 .../expected/server_echo.parsed--lowered.diff | 10 -- .../expected/server_echo.parsed.ddp | 2 - .../server_echo.projected--as-of.diff | 0 .../plan_pairs/expected/server_producer.ddp | 5 - .../server_producer.lowered--optimized.diff | 0 .../server_producer.optimized--projected.diff | 0 .../server_producer.parsed--lowered.diff | 9 -- .../expected/server_producer.parsed.ddp | 1 - .../server_producer.projected--as-of.diff | 0 .../expected/server_reach_import.ddp | 17 --- ...erver_reach_import.lowered--optimized.diff | 17 --- ...ver_reach_import.optimized--projected.diff | 0 .../server_reach_import.parsed--lowered.diff | 23 ---- .../expected/server_reach_import.parsed.ddp | 7 - .../server_reach_import.projected--as-of.diff | 0 interactive/tests/plan_pairs/manifest.rs | 5 - .../{examples => tests}/programs/adt.ddp | 0 .../{examples => tests}/programs/adt.expected | 0 .../{examples => tests}/programs/ast.ddp | 0 .../{examples => tests}/programs/ast.expected | 0 .../{examples => tests}/programs/binders.ddp | 0 .../programs/binders.expected | 0 .../{examples => tests}/programs/kcore.ddp | 0 .../programs/kcore.expected | 0 .../{examples => tests}/programs/reach.ddp | 0 .../programs/reach.expected | 0 .../{examples => tests}/programs/scc.ddp | 0 .../{examples => tests}/programs/scc.expected | 0 .../{examples => tests}/programs/stable.ddp | 0 .../programs/stable.expected | 0 .../{examples => tests}/programs/tour.ddp | 0 .../programs/tour.expected | 0 .../{examples => tests}/programs/unnest.ddp | 0 .../programs/unnest.expected | 0 interactive/tests/shared_arrangements.rs | 2 +- 104 files changed, 159 insertions(+), 1458 deletions(-) create mode 100644 interactive/examples/README.md rename interactive/examples/{server/programs => }/access.ddp (100%) rename interactive/examples/{server/sessions => }/access.txt (86%) create mode 100644 interactive/examples/components.ddp delete mode 100644 interactive/examples/recipes/README.md delete mode 100644 interactive/examples/recipes/connected_components.ddp delete mode 100644 interactive/examples/recipes/connected_components.out delete mode 100644 interactive/examples/recipes/connected_components.txt delete mode 100644 interactive/examples/recipes/left_join.ddp delete mode 100644 interactive/examples/recipes/left_join.out delete mode 100644 interactive/examples/recipes/left_join.txt delete mode 100644 interactive/examples/recipes/retention.ddp delete mode 100644 interactive/examples/recipes/retention.out delete mode 100644 interactive/examples/recipes/retention.txt delete mode 100644 interactive/examples/recipes/shortest_paths.ddp delete mode 100644 interactive/examples/recipes/shortest_paths.out delete mode 100644 interactive/examples/recipes/shortest_paths.txt delete mode 100644 interactive/examples/recipes/sum.ddp delete mode 100644 interactive/examples/recipes/sum.out delete mode 100644 interactive/examples/recipes/sum.txt delete mode 100644 interactive/examples/recipes/top_k.ddp delete mode 100644 interactive/examples/recipes/top_k.out delete mode 100644 interactive/examples/recipes/top_k.txt delete mode 100644 interactive/examples/recipes/triangles.out delete mode 100644 interactive/examples/recipes/triangles.txt delete mode 100644 interactive/examples/recipes/upsert.ddp delete mode 100644 interactive/examples/recipes/upsert.out delete mode 100644 interactive/examples/recipes/upsert.txt create mode 100644 interactive/examples/retention.ddp delete mode 100644 interactive/examples/server/README.md delete mode 100644 interactive/examples/server/programs/arrange_idem.ddp delete mode 100644 interactive/examples/server/programs/echo.ddp delete mode 100644 interactive/examples/server/programs/producer.ddp delete mode 100644 interactive/examples/server/programs/reach_import.ddp delete mode 100644 interactive/examples/server/sessions/arrange_idem.txt delete mode 100644 interactive/examples/server/sessions/drop.txt delete mode 100644 interactive/examples/server/sessions/peek.txt delete mode 100644 interactive/examples/server/sessions/shared_trace.txt delete mode 100644 interactive/examples/server/sessions/values.txt create mode 100644 interactive/examples/shortest_paths.ddp create mode 100644 interactive/examples/top_k.ddp rename interactive/examples/{recipes => }/triangles.ddp (64%) create mode 100644 interactive/examples/upsert.ddp delete mode 100644 interactive/server/tests/recipes.rs delete mode 100644 interactive/tests/plan_pairs/expected/server_access.ddp delete mode 100644 interactive/tests/plan_pairs/expected/server_access.lowered--optimized.diff delete mode 100644 interactive/tests/plan_pairs/expected/server_access.optimized--projected.diff delete mode 100644 interactive/tests/plan_pairs/expected/server_access.parsed--lowered.diff delete mode 100644 interactive/tests/plan_pairs/expected/server_access.parsed.ddp delete mode 100644 interactive/tests/plan_pairs/expected/server_access.projected--as-of.diff delete mode 100644 interactive/tests/plan_pairs/expected/server_arrange_idem.ddp delete mode 100644 interactive/tests/plan_pairs/expected/server_arrange_idem.lowered--optimized.diff delete mode 100644 interactive/tests/plan_pairs/expected/server_arrange_idem.optimized--projected.diff delete mode 100644 interactive/tests/plan_pairs/expected/server_arrange_idem.parsed--lowered.diff delete mode 100644 interactive/tests/plan_pairs/expected/server_arrange_idem.parsed.ddp delete mode 100644 interactive/tests/plan_pairs/expected/server_arrange_idem.projected--as-of.diff delete mode 100644 interactive/tests/plan_pairs/expected/server_echo.ddp delete mode 100644 interactive/tests/plan_pairs/expected/server_echo.lowered--optimized.diff delete mode 100644 interactive/tests/plan_pairs/expected/server_echo.optimized--projected.diff delete mode 100644 interactive/tests/plan_pairs/expected/server_echo.parsed--lowered.diff delete mode 100644 interactive/tests/plan_pairs/expected/server_echo.parsed.ddp delete mode 100644 interactive/tests/plan_pairs/expected/server_echo.projected--as-of.diff delete mode 100644 interactive/tests/plan_pairs/expected/server_producer.ddp delete mode 100644 interactive/tests/plan_pairs/expected/server_producer.lowered--optimized.diff delete mode 100644 interactive/tests/plan_pairs/expected/server_producer.optimized--projected.diff delete mode 100644 interactive/tests/plan_pairs/expected/server_producer.parsed--lowered.diff delete mode 100644 interactive/tests/plan_pairs/expected/server_producer.parsed.ddp delete mode 100644 interactive/tests/plan_pairs/expected/server_producer.projected--as-of.diff delete mode 100644 interactive/tests/plan_pairs/expected/server_reach_import.ddp delete mode 100644 interactive/tests/plan_pairs/expected/server_reach_import.lowered--optimized.diff delete mode 100644 interactive/tests/plan_pairs/expected/server_reach_import.optimized--projected.diff delete mode 100644 interactive/tests/plan_pairs/expected/server_reach_import.parsed--lowered.diff delete mode 100644 interactive/tests/plan_pairs/expected/server_reach_import.parsed.ddp delete mode 100644 interactive/tests/plan_pairs/expected/server_reach_import.projected--as-of.diff rename interactive/{examples => tests}/programs/adt.ddp (100%) rename interactive/{examples => tests}/programs/adt.expected (100%) rename interactive/{examples => tests}/programs/ast.ddp (100%) rename interactive/{examples => tests}/programs/ast.expected (100%) rename interactive/{examples => tests}/programs/binders.ddp (100%) rename interactive/{examples => tests}/programs/binders.expected (100%) rename interactive/{examples => tests}/programs/kcore.ddp (100%) rename interactive/{examples => tests}/programs/kcore.expected (100%) rename interactive/{examples => tests}/programs/reach.ddp (100%) rename interactive/{examples => tests}/programs/reach.expected (100%) rename interactive/{examples => tests}/programs/scc.ddp (100%) rename interactive/{examples => tests}/programs/scc.expected (100%) rename interactive/{examples => tests}/programs/stable.ddp (100%) rename interactive/{examples => tests}/programs/stable.expected (100%) rename interactive/{examples => tests}/programs/tour.ddp (100%) rename interactive/{examples => tests}/programs/tour.expected (100%) rename interactive/{examples => tests}/programs/unnest.ddp (100%) rename interactive/{examples => tests}/programs/unnest.expected (100%) diff --git a/interactive/LANGUAGE.md b/interactive/LANGUAGE.md index cf7e25abb2..8bd596b1f5 100644 --- a/interactive/LANGUAGE.md +++ b/interactive/LANGUAGE.md @@ -2,7 +2,7 @@ DDIR programs describe collections that change over time, and the server keeps every collection up to date as its inputs change. This page is the reference: what a program is made of, every collection operator, the scalar language inside operators, time, iteration, and the positions the language takes on purpose. -For worked examples of the idioms that make programs fast or correct, see the [recipes](examples/recipes/README.md). +For programs at work in the server, see the [examples](examples/README.md). For the commands that load programs, feed them and read them, see the [server reference](server/README.md#server-command-reference). ## A first program @@ -17,15 +17,15 @@ output "total" = sales | sum; A session loads it, feeds rows at times, closes those times, and reads the result: ``` -load sum from examples/recipes/sum.ddp -feed sum 0 1 val=5 time=0 -feed sum 0 2 val=40 time=0 -seal sum 0 1 -peek sum::total +load sales from sales.ddp +feed sales 0 1 val=5 time=0 +feed sales 0 2 val=40 time=0 +seal sales 0 1 +peek sales::total ``` Later feeds at later times update `total`; nothing is recomputed from scratch. -This program is the [`sum`](examples/recipes/sum.ddp) recipe. +[`examples/access.txt`](examples/access.txt) is a whole session. ## Collections @@ -124,7 +124,7 @@ The `time` is in the shape of the scope's time (see [Time](#time-and-scopes)), a A `filter` keeps the rows where `mask` is a nonzero `int`. Inside a nested scope each `0` above is a time of every coordinate, such as `(0, 0)` one level down. -`delay` retracts rows after a fixed time ([`retention`](examples/recipes/retention.ddp)) and orders work within a loop ([`connected_components`](examples/recipes/connected_components.ddp), [`shortest_paths`](examples/recipes/shortest_paths.ddp)). +`delay` retracts rows after a fixed time ([`retention`](examples/retention.ddp)) and orders work within a loop ([`components`](examples/components.ddp), [`shortest_paths`](examples/shortest_paths.ddp)). `a + b` is the union of two collections, adding diffs. It is its own operator, as it reads two inputs, and `a - b` is sugar for `a + (b | negate)`. @@ -174,9 +174,9 @@ It groups rows by key. When the value is not `()`, `count` drops it first and `distinct` keys by the whole record, so neither reduce reads a key's values together. -`sum` moves each value into its diff and counts, so it costs what `count` costs. Its `lift` only hands `join_function` each row's own time, which leaves the time as it is at any depth ([`sum`](examples/recipes/sum.ddp)). +`sum` moves each value into its diff and counts, so it costs what `count` costs. Its `lift` only hands `join_function` each row's own time, which leaves the time as it is at any depth. A key whose diffs sum to zero has no row, so a total of zero is absent. -There is no `max` or top-k operator; negate the value and use `min` or `reduce(limit n, ...)`, through a tree of groups for large inputs ([`top_k`](examples/recipes/top_k.ddp)). +There is no `max` or top-k operator; negate the value and use `min` or `reduce(limit n, ...)`, through a tree of groups for large inputs ([`top_k`](examples/top_k.ddp)). ### Operators that read time @@ -189,9 +189,9 @@ None of them is sugar. | `half_join(other, strict)` | for a lifted stream `(k, (v1, t))`: each update meets `other`'s rows `(k, v2)` whose times come before its own (strictly when `strict`, a bool written `0` or `1`, is `1`), giving `(k, ((v1, v2), t2))` at the update's time, where `t2` is the later of the two times, coordinate by coordinate; `other`'s own changes produce nothing | | `half_join(other, strict, f)` | `half_join(other, strict) \| map(f)`, without building the joined rows | -`half_join` is the building block of delta joins, which never build the intermediate results of a multiway join; see the [`triangles`](examples/recipes/triangles.ddp) recipe. +`half_join` is the building block of delta joins, which never build the intermediate results of a multiway join; see the [`triangles`](examples/triangles.ddp) example. A loop's round is the last coordinate of `lift`'s time. -`lift` with `delay` makes an input row count at its own time only ([`upsert`](examples/recipes/upsert.ddp)). +`lift` with `delay` makes an input row count at its own time only ([`upsert`](examples/upsert.ddp)). ### Arrangements and inspection @@ -270,21 +270,21 @@ From outside, `name::x` is the fixed point of `x`. A scope can hold negation, reductions and further scopes. A recursion that never stops changing never finishes, and holds up every program on the server. -Prioritized iteration uses `delay` to hold some rows back to later rounds, so that work likely to be undone waits ([`connected_components`](examples/recipes/connected_components.ddp)). +Prioritized iteration uses `delay` to hold some rows back to later rounds, so that work likely to be undone waits ([`components`](examples/components.ddp)). It can make the first computation much faster, and some later edits slower. At the root, `var` and `set` carry state from one time to the next instead. A root variable is empty at first, and at each time `t` holds what its `set` held at `t - 1`. -Nothing iterates, and anything may feed the `set`, as the cycle passes through the step ([`upsert`](examples/recipes/upsert.ddp)). +Nothing iterates, and anything may feed the `set`, as the cycle passes through the step ([`upsert`](examples/upsert.ddp)). Root variables need a one-coordinate root time. ## Design positions These are deliberate, and differ from what SQL users expect. -- **Inputs are multisets.** Feeding a row twice gives it a count of 2, and retracting an absent row gives it a count of -1. `distinct` keeps any row whose count is not zero. Replacing a key's value is a program's job ([`upsert`](examples/recipes/upsert.ddp)). +- **Inputs are multisets.** Feeding a row twice gives it a count of 2, and retracting an absent row gives it a count of -1. `distinct` keeps any row whose count is not zero. Replacing a key's value is a program's job ([`upsert`](examples/upsert.ddp)). - **Arithmetic does not fail.** Integers wrap: `(9223372036854775807, 1) add` is `-9223372036854775808`. `(x, 0) div` is 0 and `(x, 0) rem` is `x`; division rounds toward zero, so `(-7, 2) div` is -3. A program that needs errors checks its operands and builds a `Result`. -- **There is no NULL.** A missing value is a variant. Declare an enum for it ([`left_join`](examples/recipes/left_join.ddp)); `() inject None` is refused, because `Option`'s `Some` lane has no shape until a value supplies one. +- **There is no NULL.** A missing value is a variant. Declare an enum for it; `() inject None` is refused, because `Option`'s `Some` lane has no shape until a value supplies one. - **Nothing is durable.** A server's programs and data last as long as its process. - **One server is one trust domain.** Programs share one command queue and one set of workers: a slow program delays the others, and a program that never finishes or crashes a worker stops them all. - **Inputs belong to their program.** Another program cannot read an input, only an output. Changing a program's schema is a migration; a stable program that only publishes its inputs, read by programs whose inputs are bound to them, keeps views replaceable. diff --git a/interactive/README.md b/interactive/README.md index a89614d802..eca3ceccc4 100644 --- a/interactive/README.md +++ b/interactive/README.md @@ -1,6 +1,6 @@ # DDIR -[Language reference](LANGUAGE.md) · [Recipes](examples/recipes/README.md) · [Server command reference](server/README.md#server-command-reference) +[Examples](examples/README.md) · [Language reference](LANGUAGE.md) · [Server command reference](server/README.md#server-command-reference) An interpreted language (and runtime) for computation over continually changing collections. @@ -177,12 +177,12 @@ The intent is that the operation of the system is conducted less by humans and k The examples will feed and read data with textual, row-oriented versions. This is better done through binary columnar forms, which exist but aren't easy to present. -The program is [`examples/server/programs/access.ddp`](examples/server/programs/access.ddp), and the whole session is [`examples/server/sessions/access.txt`](examples/server/sessions/access.txt). -From `interactive/`, `cargo run --release -p ddir-server < examples/server/sessions/access.txt` runs it end to end. +The program is [`examples/access.ddp`](examples/access.ddp), and the whole session is [`examples/access.txt`](examples/access.txt). +From `interactive/`, `cargo run --release -p ddir-server < examples/access.txt` runs it end to end. ``` # Install access.ddp under the name `demo`. -load demo from examples/server/programs/access.ddp +load demo from examples/access.ddp ``` This binds the name `demo` to the root scope of `access.ddp`. There are now three inputs we can reference, and the output `access`. @@ -329,26 +329,18 @@ The stack aims to stay generic over two things: the expression language and the Another expression language or backend should plug in at those two places, without the rest of the stack depending on it. -The `examples/programs/` directory contains example programs, intentionally simple at the moment. The one executable is the server (the `ddir-server` crate in `server/`, documented in -`server/README.md`; the session scripts in `examples/server/` show it at work); a program runs by -loading it, feeding its inputs at a time, and sealing them past that time. For example, feed a two-edge path and its root on four workers. +`server/README.md`); a program runs by loading it, feeding its inputs at a time, and sealing +them past that time. The session in [`examples/`](examples/README.md) shows it at work. For +example, run the access session on four workers: ``` cd interactive -printf 'load reach from examples/programs/reach.ddp -feed reach 0 0,1 time=0 -feed reach 0 1,2 time=0 -feed reach 1 0 time=0 -seal reach 0 1 -seal reach 1 1 -peek reach::result as-of 0 -exit -' | DDIR_WORKERS=4 cargo run --release -p ddir-server +DDIR_WORKERS=4 cargo run --release -p ddir-server < examples/access.txt ``` For the full command vocabulary — `load`, `feed`, `seal`, `peek`, `tail`, and the rest — see -[`examples/server/README.md`](examples/server/README.md), and `server/README.md` for the -protocol and configuration. +[`server/README.md`](server/README.md#server-command-reference), which also has the protocol +and configuration. ## Differential testing diff --git a/interactive/examples/README.md b/interactive/examples/README.md new file mode 100644 index 0000000000..56cf9226d4 --- /dev/null +++ b/interactive/examples/README.md @@ -0,0 +1,13 @@ +# Examples + +[`access.txt`](access.txt) is a whole session: it loads [`access.ddp`](access.ddp), feeds it, and reads the results. +Run it from `interactive/`, and see [the README](../README.md#operation) for a walk through it: + +```sh +cargo run --release -p ddir-server < examples/access.txt +``` + +The other programs each show one idiom, explained in their comments: +[`top_k`](top_k.ddp), [`retention`](retention.ddp), [`upsert`](upsert.ddp), [`components`](components.ddp), [`shortest_paths`](shortest_paths.ddp), [`triangles`](triangles.ddp). + +[`aoc2023`](aoc2023/README.md) is Advent of Code 2023 in DDIR, with its own runner. diff --git a/interactive/examples/server/programs/access.ddp b/interactive/examples/access.ddp similarity index 100% rename from interactive/examples/server/programs/access.ddp rename to interactive/examples/access.ddp diff --git a/interactive/examples/server/sessions/access.txt b/interactive/examples/access.txt similarity index 86% rename from interactive/examples/server/sessions/access.txt rename to interactive/examples/access.txt index 90fd75c628..dc9127905a 100644 --- a/interactive/examples/server/sessions/access.txt +++ b/interactive/examples/access.txt @@ -1,8 +1,8 @@ -# The access-control example from interactive/README.md. Run from the interactive/ dir: -# cargo run --release -p ddir-server < examples/server/sessions/access.txt +# The access-control session that interactive/README.md walks through. +# Run from interactive/: cargo run --release -p ddir-server < examples/access.txt # Install access.ddp under the name `demo`. -load demo from examples/server/programs/access.ddp +load demo from examples/access.ddp # Variants are structural, and written by position: # inject(0,x) is User x, inject(1,x) is Group x, diff --git a/interactive/examples/components.ddp b/interactive/examples/components.ddp new file mode 100644 index 0000000000..9072557c50 --- /dev/null +++ b/interactive/examples/components.ddp @@ -0,0 +1,19 @@ +-- Connected components by label propagation, with prioritized iteration. +-- +-- Each node takes the least label among its neighbors. Small labels win, so `delay` holds node +-- n's own label back to round 256 * bit_length(n), letting small labels spread before large ones +-- start work that would be undone. On a 1M-edge graph this took 1.39 s to 0.55 s; some later +-- edits got slower (a bridge in a 300-node path: 37 ms to 3.4 s). + +let raw = input "edges" : (int, int); -- an undirected edge, either way round +let edges = raw + (raw | map((a, b) -> (b, a))) | distinct; +let nodes = edges | map((a, _) -> (a, ())) | distinct; + +cc: { + -- The inner time is (epoch, round): `0` leaves the epoch alone and sets the round. + let seeds = nodes | map((n, _) -> (n, n)) | delay((_, n) -> (0, (256, n call bit_length) mul)); + let proposals = labels | join(edges, (_, (label, dst)) -> (dst, label)); + set labels = seeds + proposals | min; +} + +output "labels" = cc::labels; diff --git a/interactive/examples/recipes/README.md b/interactive/examples/recipes/README.md deleted file mode 100644 index eadb765a1a..0000000000 --- a/interactive/examples/recipes/README.md +++ /dev/null @@ -1,30 +0,0 @@ -# Recipes - -Each recipe is a small program for one idiom, with a session that runs it on a few rows. -The idioms are the ones that make DDIR programs fast or correct, and that are hard to find from the operator list alone. - -Run a recipe from `interactive/`, and compare with its recorded transcript: - -```sh -cargo run --release -p ddir-server < examples/recipes/upsert.txt -``` - -`NAME.ddp` is the program and explains the idiom in its comments. -`NAME.txt` feeds it, seals its inputs, and peeks the results. -`NAME.out` is the transcript, checked at one and three workers by `server/tests/recipes.rs`. -After an intended change, `UPDATE_RECIPES=1 cargo test --release -p ddir-server --test recipes` records the transcripts again. - -| Recipe | Idiom | Use it for | -|---|---|---| -| [`sum`](sum.ddp) | `sum` adds each key's values, weighted by their diffs | sums of integers, incrementally | -| [`top_k`](top_k.ddp) | a tree of `reduce(limit k)` groups | top k over many rows, with small updates | -| [`retention`](retention.ddp) | `lift` and `delay` retract each row a fixed time after it arrives | sliding windows, expiry | -| [`upsert`](upsert.ddp) | a root `var` holds the table one time ago; writes count at their own time only | a keyed table with put and delete | -| [`left_join`](left_join.ddp) | the unmatched rows are the input minus the matched ones; a declared `enum` for the missing value | outer joins, without NULL | -| [`connected_components`](connected_components.ddp) | `delay` holds back large labels, so small ones spread first | label propagation, prioritized iteration | -| [`shortest_paths`](shortest_paths.ddp) | `delay` to bucket `d / Δ`, spaced several rounds apart | weighted shortest paths, delta stepping | -| [`triangles`](triangles.ddp) | `lift` and `half_join` along each use of a relation | multiway joins without large intermediate results | - -Two facts the recipes lean on: -- **Inputs are multisets.** A row's count is the sum of its diffs, so a second put of a key adds a second row; replacing a value is a program's job, as in `upsert`. -- **`(e)` is a one-field tuple, not `e`.** Parentheses build a tuple. Write `t` where you mean the value itself; `(t)` is a different shape. diff --git a/interactive/examples/recipes/connected_components.ddp b/interactive/examples/recipes/connected_components.ddp deleted file mode 100644 index 5f99425c69..0000000000 --- a/interactive/examples/recipes/connected_components.ddp +++ /dev/null @@ -1,22 +0,0 @@ --- Connected components by label propagation, with prioritized iteration. --- --- Each node starts labeled by itself, and repeatedly takes the least label among its neighbors. --- Unprioritized, every node's own label starts spreading in the first round, and most of that --- work is undone when a smaller label arrives. Here `delay` holds node n's own label back to round --- 256 * bit_length(n), so small labels spread before large ones enter; `scc.ddp` does the same. --- On a 1M-edge graph this took the first computation from 1.39 s to 0.55 s. --- The cost moves to some later edits: deleting and re-adding a bridge to a long path took --- longer with the delay than without (0.7 s and 3.4 s against 37 ms, for a 300-node path). - -let raw = input 0 : (int, int); -- an undirected edge, either way round -let edges = raw + (raw | map((a, b) -> (b, a))) | distinct; -let nodes = edges | map((a, _) -> (a, ())) | distinct; - -cc: { - -- The inner time is (epoch, round): `0` leaves the epoch alone and sets the round. - let seeds = nodes | map((n, _) -> (n, n)) | delay((_, n) -> (0, (256, n call bit_length) mul)); - let proposals = labels | join(edges, (_, (label, dst)) -> (dst, label)); - set labels = seeds + proposals | min; -} - -output "labels" = cc::labels; diff --git a/interactive/examples/recipes/connected_components.out b/interactive/examples/recipes/connected_components.out deleted file mode 100644 index 11d84be088..0000000000 --- a/interactive/examples/recipes/connected_components.out +++ /dev/null @@ -1,27 +0,0 @@ -_1 ok installed "cc" -_2 ok fed "cc" input 0 at t=0 -_3 ok fed "cc" input 0 at t=0 -_4 ok fed "cc" input 0 at t=0 -_5 ok sealed "cc" input 0 to 1 -_6 data 1 val=1 -_6 data 2 val=1 -_6 data 3 val=1 -_6 data 4 val=4 -_6 data 5 val=4 -_6 ok as-of=0 -_7 ok fed "cc" input 0 at t=1 -_8 ok sealed "cc" input 0 to 2 -_9 data 1 val=1 -_9 data 2 val=1 -_9 data 3 val=1 -_9 data 4 val=1 -_9 data 5 val=1 -_9 ok as-of=1 -_10 ok fed "cc" input 0 at t=2 -_11 ok sealed "cc" input 0 to 3 -_12 data 2 val=2 -_12 data 3 val=2 -_12 data 4 val=2 -_12 data 5 val=2 -_12 ok as-of=2 -_13 ok bye diff --git a/interactive/examples/recipes/connected_components.txt b/interactive/examples/recipes/connected_components.txt deleted file mode 100644 index 90ed17f3db..0000000000 --- a/interactive/examples/recipes/connected_components.txt +++ /dev/null @@ -1,21 +0,0 @@ -# Connected components with prioritized label propagation: see connected_components.ddp. -# Run from interactive/: ddir_server < examples/recipes/connected_components.txt -load cc from examples/recipes/connected_components.ddp - -# Time 0: two components, {1, 2, 3} and {4, 5}. -feed cc 0 1 val=2 time=0 -feed cc 0 3 val=2 time=0 -feed cc 0 5 val=4 time=0 -seal cc 0 1 -peek cc::labels - -# Time 1: the edge 3-4 joins them. -feed cc 0 3 val=4 time=1 -seal cc 0 2 -peek cc::labels - -# Time 2: removing 1-2 leaves node 1 with no edges, so no label; the rest take label 2. -feed cc 0 1 val=2 time=2 diff=-1 -seal cc 0 3 -peek cc::labels -exit diff --git a/interactive/examples/recipes/left_join.ddp b/interactive/examples/recipes/left_join.ddp deleted file mode 100644 index 92b1e628a7..0000000000 --- a/interactive/examples/recipes/left_join.ddp +++ /dev/null @@ -1,17 +0,0 @@ --- A left join: every order, with its customer's name when there is one. --- --- There is no NULL; a missing value is a variant. Declaring the enum gives each variant its --- payload shape, so `() inject Unknown` needs nothing more. (The built-in `Option` cannot yet --- build `None` this way: its `Some` lane has no shape until a value supplies one.) --- The unmatched rows are the input minus the ones the join matched. - -enum Name = Known List(int) | Unknown; - -let orders = input 0 : (int, int); -- (order, customer) -let customers = input 1 : (int, List(int)); -- (customer, name); a string is a list of bytes - -let by_customer = orders | map((order, customer) -> (customer, order)); -let matched = by_customer | join(customers, (c, (order, name)) -> (order, (c, name inject Known))); -let unmatched = by_customer - (by_customer | join(customers, (c, (order, _)) -> (c, order))); - -output "orders" = matched + (unmatched | map((c, order) -> (order, (c, () inject Unknown)))); diff --git a/interactive/examples/recipes/left_join.out b/interactive/examples/recipes/left_join.out deleted file mode 100644 index 94d55615ee..0000000000 --- a/interactive/examples/recipes/left_join.out +++ /dev/null @@ -1,16 +0,0 @@ -_1 ok installed "lj" -_2 ok fed "lj" input 1 at t=0 -_3 ok fed "lj" input 0 at t=0 -_4 ok fed "lj" input 0 at t=0 -_5 ok sealed "lj" input 0 to 1 -_6 ok sealed "lj" input 1 to 1 -_7 data 100 val=tuple(1,inject(0,list(97,110,110))) -_7 data 101 val=tuple(2,inject(1,tuple())) -_7 ok as-of=0 -_8 ok fed "lj" input 1 at t=1 -_9 ok sealed "lj" input 0 to 2 -_10 ok sealed "lj" input 1 to 2 -_11 data 100 val=tuple(1,inject(0,list(97,110,110))) -_11 data 101 val=tuple(2,inject(0,list(98,111))) -_11 ok as-of=1 -_12 ok bye diff --git a/interactive/examples/recipes/left_join.txt b/interactive/examples/recipes/left_join.txt deleted file mode 100644 index 27bab4ffb0..0000000000 --- a/interactive/examples/recipes/left_join.txt +++ /dev/null @@ -1,19 +0,0 @@ -# A left join with a declared enum for the missing side: see left_join.ddp. -# Run from interactive/: ddir_server < examples/recipes/left_join.txt -load lj from examples/recipes/left_join.ddp - -# A feed spells a string as its bytes: "ann" is list(97,110,110). -feed lj 1 1 val=list(97,110,110) time=0 -feed lj 0 100 val=1 time=0 -feed lj 0 101 val=2 time=0 -seal lj 0 1 -seal lj 1 1 -# Order 101's customer is unknown: `inject(1,tuple())`. Order 100's is `inject(0,)`. -peek lj::orders - -# Customer 2 arrives, and order 101 moves from Unknown to Known. -feed lj 1 2 val=list(98,111) time=1 -seal lj 0 2 -seal lj 1 2 -peek lj::orders -exit diff --git a/interactive/examples/recipes/retention.ddp b/interactive/examples/recipes/retention.ddp deleted file mode 100644 index 636ba58a40..0000000000 --- a/interactive/examples/recipes/retention.ddp +++ /dev/null @@ -1,13 +0,0 @@ --- A sliding window: count each key's events over the last 3 times. --- --- `lift` pairs each row with its time t, and `delay` moves a copy of the row to t + 3. --- Subtracting that copy retracts each event 3 times after it arrived, so the window holds only --- recent events, and its state stays the size of the window rather than of the whole history. --- Joining against a clock input instead would redo the work for every event at every tick. - -let events = input 0 : (int, ()); -- an event for a key, at the time it is fed - -let expired = events | lift | delay((_, (_, (t))) -> (t, 3) add) | map((k, _) -> (k, ())); -let recent = events - expired; - -output "recent" = recent | count; diff --git a/interactive/examples/recipes/retention.out b/interactive/examples/recipes/retention.out deleted file mode 100644 index c440ea3992..0000000000 --- a/interactive/examples/recipes/retention.out +++ /dev/null @@ -1,16 +0,0 @@ -_1 ok installed "win" -_2 ok fed "win" input 0 at t=0 -_3 ok fed "win" input 0 at t=1 -_4 ok fed "win" input 0 at t=1 -_5 ok fed "win" input 0 at t=2 -_6 ok sealed "win" input 0 to 3 -_7 data 7 val=3 -_7 data 8 val=1 -_7 ok as-of=2 -_8 ok sealed "win" input 0 to 5 -_9 data 7 val=2 -_9 data 8 val=1 -_9 ok as-of=3 -_10 data 7 val=1 -_10 ok as-of=4 -_11 ok bye diff --git a/interactive/examples/recipes/retention.txt b/interactive/examples/recipes/retention.txt deleted file mode 100644 index b3bfdedcfc..0000000000 --- a/interactive/examples/recipes/retention.txt +++ /dev/null @@ -1,17 +0,0 @@ -# A sliding window with `delay`: see retention.ddp. -# Run from interactive/: ddir_server < examples/recipes/retention.txt -load win from examples/recipes/retention.ddp - -feed win 0 7 time=0 -feed win 0 7 time=1 -feed win 0 8 time=1 -feed win 0 7 time=2 -seal win 0 3 -# Times 0, 1 and 2 are all in the window at time 2. -peek win::recent - -# At time 3 the event from time 0 leaves the window; at time 4, those from time 1 leave. -seal win 0 5 -peek win::recent as-of 3 -peek win::recent as-of 4 -exit diff --git a/interactive/examples/recipes/shortest_paths.ddp b/interactive/examples/recipes/shortest_paths.ddp deleted file mode 100644 index c2510fc824..0000000000 --- a/interactive/examples/recipes/shortest_paths.ddp +++ /dev/null @@ -1,22 +0,0 @@ --- Single-source shortest paths with delta stepping. --- --- Plain relaxation in a loop finds every path, but on weighted graphs it keeps improving long --- tentative distances that a shorter path later replaces. Delta stepping settles distances in --- buckets of width Δ, nearest first. Here `delay` puts a proposed distance d at round --- (d / 64) * 16: bucket d / 64, spaced 16 rounds apart, so a bucket's hop-by-hop relaxations --- finish before the next bucket starts. With the spacing at 1 a bucket's relaxations run into --- the next bucket's and the delay gains nothing. --- On a 500 x 500 grid with weights 1 to 100 this took the first computation from 64.5 s to 0.9 s. --- Choose Δ near the typical edge weight, and the spacing above the hops a bucket needs. - -let edges = input 0 : (int, (int, int)); -- (src, (dst, weight)) -let roots = input 1 : (int, ()); -- the sources, at distance 0 - -sssp: { - let proposals = dist - | join(edges, (_, (d, (dst, w))) -> (dst, (d, w) add)) - | delay((_, d) -> (0, ((d, 64) div, 16) mul)); - set dist = (roots | map((n, ()) -> (n, 0))) + proposals | min; -} - -output "dist" = sssp::dist; diff --git a/interactive/examples/recipes/shortest_paths.out b/interactive/examples/recipes/shortest_paths.out deleted file mode 100644 index 246a482f58..0000000000 --- a/interactive/examples/recipes/shortest_paths.out +++ /dev/null @@ -1,23 +0,0 @@ -_1 ok installed "sp" -_2 ok fed "sp" input 0 at t=0 -_3 ok fed "sp" input 0 at t=0 -_4 ok fed "sp" input 0 at t=0 -_5 ok fed "sp" input 0 at t=0 -_6 ok fed "sp" input 1 at t=0 -_7 ok sealed "sp" input 0 to 1 -_8 ok sealed "sp" input 1 to 1 -_9 data 1 val=0 -_9 data 2 val=200 -_9 data 3 val=400 -_9 data 4 val=410 -_9 ok as-of=0 -_10 ok fed "sp" input 0 at t=1 -_11 ok fed "sp" input 0 at t=1 -_12 ok sealed "sp" input 0 to 2 -_13 ok sealed "sp" input 1 to 2 -_14 data 1 val=0 -_14 data 2 val=200 -_14 data 3 val=500 -_14 data 4 val=510 -_14 ok as-of=1 -_15 ok bye diff --git a/interactive/examples/recipes/shortest_paths.txt b/interactive/examples/recipes/shortest_paths.txt deleted file mode 100644 index 2ec40885ee..0000000000 --- a/interactive/examples/recipes/shortest_paths.txt +++ /dev/null @@ -1,21 +0,0 @@ -# Shortest paths with delta stepping: see shortest_paths.ddp. -# Run from interactive/: ddir_server < examples/recipes/shortest_paths.txt -load sp from examples/recipes/shortest_paths.ddp - -# Time 0: 1 -> 2 -> 3 costs 200 + 200; the direct edge 1 -> 3 costs 500. -feed sp 0 1 val=tuple(2,200) time=0 -feed sp 0 2 val=tuple(3,200) time=0 -feed sp 0 1 val=tuple(3,500) time=0 -feed sp 0 3 val=tuple(4,10) time=0 -feed sp 1 1 time=0 -seal sp 0 1 -seal sp 1 1 -peek sp::dist - -# Time 1: the edge 2 -> 3 gets dearer, and the direct edge wins. -feed sp 0 2 val=tuple(3,200) time=1 diff=-1 -feed sp 0 2 val=tuple(3,900) time=1 -seal sp 0 2 -seal sp 1 2 -peek sp::dist -exit diff --git a/interactive/examples/recipes/sum.ddp b/interactive/examples/recipes/sum.ddp deleted file mode 100644 index 0d1969ca04..0000000000 --- a/interactive/examples/recipes/sum.ddp +++ /dev/null @@ -1,9 +0,0 @@ --- Sum an integer per key with `sum`, which adds each key's values weighted by their diffs. --- --- `sum` is sugar: it moves each value into its row's diff, and `count` adds up the diffs. --- The sum is maintained incrementally, where folding a list of values would rebuild it per change. --- A total of zero has no row, like any record whose diffs cancel; it is bounded by `i64`. - -let sales = input 0 : (int, int); -- (store, amount); a refund is a negative amount - -output "total" = sales | sum; diff --git a/interactive/examples/recipes/sum.out b/interactive/examples/recipes/sum.out deleted file mode 100644 index 1b660ce9ee..0000000000 --- a/interactive/examples/recipes/sum.out +++ /dev/null @@ -1,15 +0,0 @@ -_1 ok installed "sum" -_2 ok fed "sum" input 0 at t=0 -_3 ok fed "sum" input 0 at t=0 -_4 ok fed "sum" input 0 at t=0 -_5 ok fed "sum" input 0 at t=0 -_6 ok sealed "sum" input 0 to 1 -_7 data 1 val=19 -_7 data 2 val=40 -_7 ok as-of=0 -_8 ok fed "sum" input 0 at t=1 -_9 ok fed "sum" input 0 at t=1 -_10 ok sealed "sum" input 0 to 2 -_11 data 1 val=20 -_11 ok as-of=1 -_12 ok bye diff --git a/interactive/examples/recipes/sum.txt b/interactive/examples/recipes/sum.txt deleted file mode 100644 index c93ddd3f6b..0000000000 --- a/interactive/examples/recipes/sum.txt +++ /dev/null @@ -1,18 +0,0 @@ -# Sum per key with `sum`: see sum.ddp. -# Run from interactive/: ddir_server < examples/recipes/sum.txt -load sum from examples/recipes/sum.ddp - -# Time 0: three sales at store 1, one at store 2. -feed sum 0 1 val=5 time=0 -feed sum 0 1 val=7 time=0 -feed sum 0 1 val=7 time=0 -feed sum 0 2 val=40 time=0 -seal sum 0 1 -peek sum::total - -# Time 1: a refund at store 2 brings it to zero, and its row goes away. -feed sum 0 2 val=-40 time=1 -feed sum 0 1 val=1 time=1 -seal sum 0 2 -peek sum::total -exit diff --git a/interactive/examples/recipes/top_k.ddp b/interactive/examples/recipes/top_k.ddp deleted file mode 100644 index c4589c8e37..0000000000 --- a/interactive/examples/recipes/top_k.ddp +++ /dev/null @@ -1,19 +0,0 @@ --- The top 3 scores overall, maintained as scores change. --- --- `reduce(limit 3, ...)` hands its lambda only a key's first 3 values in value order, so storing --- the negated score puts the highest first. One group holding every player would redo that --- group's work on every update; a tree of groups keeps each update small. Here players fall into --- 64 buckets, each bucket keeps its own top 3, and the root takes the top 3 of those. --- An update then touches one bucket and the 64 * 3 candidates at the root. --- Ties order by player, the second field of the value. - -let scores = input 0 : (int, int); -- (player, score) - -let buckets = scores - | map((player, score) -> ((player, 64) rem, (score neg, player))) - | reduce(limit 3, (_, vals) -> vals); -let top = buckets - | map((_, v) -> ((), v)) - | reduce(limit 3, (_, vals) -> vals); - -output "top" = top | map((_, (negated, player)) -> (player, negated neg)); diff --git a/interactive/examples/recipes/top_k.out b/interactive/examples/recipes/top_k.out deleted file mode 100644 index 2222a08a39..0000000000 --- a/interactive/examples/recipes/top_k.out +++ /dev/null @@ -1,21 +0,0 @@ -_1 ok installed "top" -_2 ok fed "top" input 0 at t=0 -_3 ok fed "top" input 0 at t=0 -_4 ok fed "top" input 0 at t=0 -_5 ok fed "top" input 0 at t=0 -_6 ok fed "top" input 0 at t=0 -_7 ok sealed "top" input 0 to 1 -_8 data 2 val=80 -_8 data 65 val=70 -_8 data 129 val=90 -_8 ok as-of=0 -_9 ok fed "top" input 0 at t=1 -_10 ok fed "top" input 0 at t=1 -_11 ok fed "top" input 0 at t=1 -_12 ok fed "top" input 0 at t=1 -_13 ok sealed "top" input 0 to 2 -_14 data 2 val=80 -_14 data 3 val=95 -_14 data 65 val=70 -_14 ok as-of=1 -_15 ok bye diff --git a/interactive/examples/recipes/top_k.txt b/interactive/examples/recipes/top_k.txt deleted file mode 100644 index ef7b655c79..0000000000 --- a/interactive/examples/recipes/top_k.txt +++ /dev/null @@ -1,22 +0,0 @@ -# Top k through a tree of `reduce(limit k)`: see top_k.ddp. -# Run from interactive/: ddir_server < examples/recipes/top_k.txt -load top from examples/recipes/top_k.ddp - -# Time 0: five players. -feed top 0 1 val=50 time=0 -feed top 0 2 val=80 time=0 -feed top 0 65 val=70 time=0 -feed top 0 3 val=10 time=0 -feed top 0 129 val=90 time=0 -seal top 0 1 -peek top::top - -# Time 1: player 3 climbs to the top, and player 129 drops out: a score change is a retraction -# of the old row and an insertion of the new one. -feed top 0 3 val=10 time=1 diff=-1 -feed top 0 3 val=95 time=1 -feed top 0 129 val=90 time=1 diff=-1 -feed top 0 129 val=20 time=1 -seal top 0 2 -peek top::top -exit diff --git a/interactive/examples/recipes/triangles.out b/interactive/examples/recipes/triangles.out deleted file mode 100644 index c10767d9c6..0000000000 --- a/interactive/examples/recipes/triangles.out +++ /dev/null @@ -1,24 +0,0 @@ -_1 ok installed "tri" -_2 ok fed "tri" input 0 at t=0 -_3 ok fed "tri" input 0 at t=0 -_4 ok fed "tri" input 0 at t=0 -_5 ok fed "tri" input 0 at t=0 -_6 ok fed "tri" input 0 at t=0 -_7 ok fed "tri" input 0 at t=0 -_8 ok fed "tri" input 0 at t=0 -_9 ok sealed "tri" input 0 to 1 -_10 data tuple() val=4 -_10 ok as-of=0 -_11 data 3 val=tuple(2,1) -_11 data 4 val=tuple(2,1) -_11 data 4 val=tuple(3,1) -_11 data 4 val=tuple(3,2) -_11 ok as-of=0 -_12 ok fed "tri" input 0 at t=1 -_13 ok sealed "tri" input 0 to 2 -_14 data tuple() val=2 -_14 ok as-of=1 -_15 data 4 val=tuple(3,1) -_15 data 4 val=tuple(3,2) -_15 ok as-of=1 -_16 ok bye diff --git a/interactive/examples/recipes/triangles.txt b/interactive/examples/recipes/triangles.txt deleted file mode 100644 index 85f78d716f..0000000000 --- a/interactive/examples/recipes/triangles.txt +++ /dev/null @@ -1,22 +0,0 @@ -# Triangles as a delta join: see triangles.ddp. -# Run from interactive/: ddir_server < examples/recipes/triangles.txt -load tri from examples/recipes/triangles.ddp - -# Time 0: all six edges among 1, 2, 3, 4 (four triangles), and the edge 4-5 (none). -feed tri 0 1 val=2 time=0 -feed tri 0 1 val=3 time=0 -feed tri 0 1 val=4 time=0 -feed tri 0 2 val=3 time=0 -feed tri 0 2 val=4 time=0 -feed tri 0 3 val=4 time=0 -feed tri 0 4 val=5 time=0 -seal tri 0 1 -peek tri::total -peek tri::triangles - -# Time 1: removing 1-2 leaves the two triangles through 3-4. -feed tri 0 1 val=2 time=1 diff=-1 -seal tri 0 2 -peek tri::total -peek tri::triangles -exit diff --git a/interactive/examples/recipes/upsert.ddp b/interactive/examples/recipes/upsert.ddp deleted file mode 100644 index f400974ccb..0000000000 --- a/interactive/examples/recipes/upsert.ddp +++ /dev/null @@ -1,25 +0,0 @@ --- Upsert: a table keyed by `int`, where a put replaces the key's value and a delete removes it. --- --- Inputs are multisets, so feeding `5 val=2` after `5 val=1` leaves both rows. --- Here the table is instead a root feedback variable: at each time it holds what `now` held --- one time earlier, and `now` is that table with this time's writes applied. --- The client never retracts anything; it only feeds the writes. - -let puts = input 0 : (int, int); -- (key, value): set key to value -let dels = input 1 : (int, ()); -- key: remove key - --- A write counts at its own time only: `lift` pairs each row with its time t, and `delay` --- moves a copy to t + 1, where subtracting it retracts the row. -let put_now = puts - (puts | lift | delay((_, (_, (t))) -> (t, 1) add) | map((k, (v, _)) -> (k, v))); -let del_now = dels - (dels | lift | delay((_, (_, (t))) -> (t, 1) add) | map((k, _) -> (k, ()))); - --- Several puts to one key at one time: the least value wins. A put beats a delete at one time. -let written = put_now | min; -let touched = (written | map((k, _) -> (k, ()))) + del_now | distinct; - -var table : (int, int); -let kept = table - (table | join(touched, (k, (v, ())) -> (k, v))); -let now = kept + written; -set table = now; - -output "table" = now; diff --git a/interactive/examples/recipes/upsert.out b/interactive/examples/recipes/upsert.out deleted file mode 100644 index e5343b8aff..0000000000 --- a/interactive/examples/recipes/upsert.out +++ /dev/null @@ -1,33 +0,0 @@ -_1 ok installed "upsert" -_2 ok fed "upsert" input 0 at t=0 -_3 ok fed "upsert" input 0 at t=0 -_4 ok sealed "upsert" input 0 to 1 -_5 ok sealed "upsert" input 1 to 1 -_6 data 1 val=10 -_6 data 2 val=20 -_6 ok as-of=0 -_7 ok fed "upsert" input 0 at t=1 -_8 ok fed "upsert" input 0 at t=1 -_9 ok sealed "upsert" input 0 to 2 -_10 ok sealed "upsert" input 1 to 2 -_11 data 1 val=11 -_11 data 2 val=20 -_11 data 3 val=30 -_11 ok as-of=1 -_12 ok fed "upsert" input 1 at t=2 -_13 ok fed "upsert" input 0 at t=2 -_14 ok fed "upsert" input 0 at t=2 -_15 ok sealed "upsert" input 0 to 3 -_16 ok sealed "upsert" input 1 to 3 -_17 data 1 val=11 -_17 data 3 val=31 -_17 ok as-of=2 -_18 ok sealed "upsert" input 0 to 4 -_19 ok sealed "upsert" input 1 to 4 -_20 data 1 val=11 -_20 data 3 val=31 -_20 ok as-of=3 -_21 data 1 val=10 -_21 data 2 val=20 -_21 ok as-of=0 -_22 ok bye diff --git a/interactive/examples/recipes/upsert.txt b/interactive/examples/recipes/upsert.txt deleted file mode 100644 index 6b06dfb60e..0000000000 --- a/interactive/examples/recipes/upsert.txt +++ /dev/null @@ -1,34 +0,0 @@ -# Upsert with a root feedback variable: see upsert.ddp. -# Run from interactive/: ddir_server < examples/recipes/upsert.txt -load upsert from examples/recipes/upsert.ddp - -# Time 0: put keys 1 and 2. -feed upsert 0 1 val=10 time=0 -feed upsert 0 2 val=20 time=0 -seal upsert 0 1 -seal upsert 1 1 -peek upsert::table - -# Time 1: overwrite key 1, and add key 3. -feed upsert 0 1 val=11 time=1 -feed upsert 0 3 val=30 time=1 -seal upsert 0 2 -seal upsert 1 2 -peek upsert::table - -# Time 2: delete key 2, and put key 3 twice: the least value wins. -feed upsert 1 2 time=2 -feed upsert 0 3 val=33 time=2 -feed upsert 0 3 val=31 time=2 -seal upsert 0 3 -seal upsert 1 3 -peek upsert::table - -# Time 3: no writes; the table stays as it was. -seal upsert 0 4 -seal upsert 1 4 -peek upsert::table - -# Reading back in time still works. -peek upsert::table as-of 0 -exit diff --git a/interactive/examples/retention.ddp b/interactive/examples/retention.ddp new file mode 100644 index 0000000000..ab078f2e63 --- /dev/null +++ b/interactive/examples/retention.ddp @@ -0,0 +1,10 @@ +-- A sliding window: each key's events over the last 3 times. +-- +-- `lift` pairs a row with its time t, and `delay` moves a copy to t + 3, where subtracting it +-- retracts the event. State stays the size of the window, and no clock input is needed. + +let events = input "events" : (int, ()); -- an event for a key + +let expired = events | lift | delay((_, (_, (t))) -> (t, 3) add) | map((k, _) -> (k, ())); + +output "recent" = events - expired | count; diff --git a/interactive/examples/server/README.md b/interactive/examples/server/README.md deleted file mode 100644 index 6c899c38c9..0000000000 --- a/interactive/examples/server/README.md +++ /dev/null @@ -1,130 +0,0 @@ -# The DDIR server - -[Server command reference](../../server/README.md#server-command-reference) - - -A live instance that hosts interpreted DDIR dataflows in one running timely -computation and lets them **share results by name**. You install dataflows one -at a time; a load may bind a dataflow's inputs to traces other dataflows `output`, so a -consumer maintains a computation incrementally over a producer's output. It is -the interpreter-driven successor to the old `dd_server` crate (which hot-loaded -compiled `.so`s); here "install" means render a DDIR program, not `dlopen`. - -The implementation is `interactive::server` (the registry + lifecycle), driven -by the `ddir-server` crate in `interactive/server/` — a long-running process -that takes commands over stdin, TCP and WebSocket (its README has the protocol -and the configuration). The session scripts here are piped into its stdin. - -## Two kinds of file (don't mix them up) - -- **`programs/*.ddp`** — DDIR *programs*: dataflow definitions you `load`. - The ones here are server-oriented (their named inputs are bound to other programs' - outputs at load); the programs in `../programs/` read numbered `input`s, which you - fill with `feed` (see "Running a program in batch" below). -- **`sessions/*.txt`** — *command scripts*: a stream of server commands - (`load`/`feed`/`seal`/…) you hand to the server. You do **not** `load` a - session; you run the server *on* it. - -## Running - -``` -cd interactive -# Run a session script (paths inside the scripts are relative to here): -cargo run --release -p ddir-server < examples/server/sessions/shared_trace.txt - -# Four workers (default: one): -DDIR_WORKERS=4 cargo run --release -p ddir-server < examples/server/sessions/shared_trace.txt - -# Or interactively: type commands; `exit` (or end of input) stops the server. -cargo run --release -p ddir-server -``` - -Every response line starts with the request's id (`_7 ok …`, `_7 data …`, -`_7 err …`); the server numbers requests that do not carry one. - -## Commands - -| command | effect | -|---|---| -| `load from [