diff --git a/interactive/LANGUAGE.md b/interactive/LANGUAGE.md new file mode 100644 index 000000000..cf7e25abb --- /dev/null +++ b/interactive/LANGUAGE.md @@ -0,0 +1,299 @@ +# The DDIR language + +DDIR programs describe collections that change over time, and the server keeps every collection up to date as its inputs change. +This page is the reference: what a program is made of, every collection operator, the scalar language inside operators, time, iteration, and the positions the language takes on purpose. +For worked examples of the idioms that make programs fast or correct, see the [recipes](examples/recipes/README.md). +For the commands that load programs, feed them and read them, see the [server reference](server/README.md#server-command-reference). + +## A first program + +``` +-- Total sales per store, kept current as sales and refunds arrive. +let sales = input 0 : (int, int); -- (store, amount) + +output "total" = sales | sum; +``` + +A session loads it, feeds rows at times, closes those times, and reads the result: + +``` +load sum from examples/recipes/sum.ddp +feed sum 0 1 val=5 time=0 +feed sum 0 2 val=40 time=0 +seal sum 0 1 +peek sum::total +``` + +Later feeds at later times update `total`; nothing is recomputed from scratch. +This program is the [`sum`](examples/recipes/sum.ddp) recipe. + +## Collections + +A collection is a multiset of rows, each a `(key, value)` pair, that changes over time. +Each change is an update: a row, the time it happens, and a signed integer diff that adds to the row's count. +A row is present at a time when the sum of its diffs up to that time is not zero. + +Operators that group rows, such as `join`, `reduce` and `min`, group them by key. +When the whole row should be the key, use the unit value: `((a, b), ())` is a row keyed by the pair `(a, b)`. + +### Shapes + +Every collection has a shape, written `(key_shape, val_shape)`. + +| Shape | Meaning | +|---|---| +| `int` | a signed 64-bit integer | +| `float` | a 64-bit float | +| `()` | unit, the tuple of no fields | +| `(a, b, ...)` | a tuple | +| `List(a)` | a list; a string is `List(int)` of its UTF-8 bytes | +| `Option(a)`, `Result(a, b)` | the built-in sums | +| a declared name | an `enum` or `type` declared in the program | + +`(int, int)` is keyed by the first integer with the second as its value. +`((int, int), ())` is keyed by both, with a unit value. + +Parentheses always build a tuple: `(int)` is a tuple of one `int`, and `(x)` is a tuple of one value. +Write `int` and `x` where you mean the value itself; the two are different shapes, and mixing them is a shape error. + +### Declarations + +``` +enum Name = Known List(int) | Unknown; -- a sum type; a variant's payload is () when omitted +type Pair = (int, int); -- an alias, erased before planning +``` + +Types are global to the program, may appear in any order, and cannot be recursive. +Aliases are not yet available inside lambdas. + +## Statements + +| Statement | Meaning | +|---|---| +| `let x = expr;` | name a collection | +| `output "x" = expr;` | publish a collection as `PROGRAM::x`; root scope only, and each name once per scope | +| `name: { ... }` | a nested scope, which iterates to a fixed point (see [Iteration](#iteration)) | +| `var x;` or `var x : (k, v);` | declare a feedback variable, optionally with its shape | +| `set x = expr;` | bind a feedback variable | +| `enum ...;`, `type ...;` | declare types | + +A `--` starts a comment to the end of the line, inside lambdas too. +Within a scope the order of statements does not matter. + +### Sources + +| Source | Meaning | +|---|---| +| `input N : (k, v)` | the input labelled `N`, with its shape | +| `input "name" : (k, v)` | the input labelled `name`, an identifier, with its shape | +| `name`, `scope::name` | a binding in this scope, an enclosing one, or a child scope's result | + +An input's label is its identity, and the order inputs are written in never matters. +Writing a label again reads the same input. +An input must declare its shape. + +A program does not name the traces it reads. +The client feeds an input by its label, or the load binds it to another program's output instead, as `load p from p.ddp edges=graph::edges`. +A bound input's shape must match the shape the output is published at. + +## Operators + +An expression is a source followed by operator stages: `source | op | op`. +`+ b` and `- b` are stages too, and each stage applies to everything to its left: `a | map(f) + b | distinct` is `((a | map(f)) + b) | distinct`. +To add a whole pipeline, parenthesize it. + +Each operator's per-row logic is a lambda, written `(pattern -> body)`; see [Lambdas](#lambdas). + +Each group of operators below has one reference form, and the other operators in the group are sugar for it. +A sugar's expansion is shown next to it, and writing the expansion gives the same result. + +### Linear operators + +The reference form is `join_function((k, v) -> list)`. +Each row becomes the lambda's `List` of `((key, value), time, diff)`, at the row's time joined with `time`, with the two diffs multiplied. +The `time` is in the shape of the scope's time (see [Time](#time-and-scopes)), and a coordinate of `0` leaves that coordinate as it is. + +| Sugar | Expansion | +|---|---| +| `map((k, v) -> (k2, v2))` | `join_function((k, v) -> ((k2, v2), 0, 1) enlist)` | +| `filter((k, v) -> mask)` | `join_function((k, v) -> (mask, ((k, v), 0, 1)) enlist filter)` | +| `flatmap((k, v) -> rows)` | `join_function((k, v) -> rows map (r -> (r, 0, 1)))` | +| `negate` | `join_function((k, v) -> ((k, v), 0, -1) enlist)` | +| `delay((k, v) -> t)` | `join_function((k, v) -> ((k, v), t, 1) enlist)` | + +A `filter` keeps the rows where `mask` is a nonzero `int`. +Inside a nested scope each `0` above is a time of every coordinate, such as `(0, 0)` one level down. + +`delay` retracts rows after a fixed time ([`retention`](examples/recipes/retention.ddp)) and orders work within a loop ([`connected_components`](examples/recipes/connected_components.ddp), [`shortest_paths`](examples/recipes/shortest_paths.ddp)). + +`a + b` is the union of two collections, adding diffs. +It is its own operator, as it reads two inputs, and `a - b` is sugar for `a + (b | negate)`. + +The operators in this group are functions of data: a result at time `t` depends only on the input's rows at `t`, not on when they arrived. + +### Joins + +The reference form is `join(other)`. +For each key it pairs every row `(k, v1)` on the left with every row `(k, v2)` on the right, giving `(k, (v1, v2))`, and multiplies their diffs. + +| Sugar | Expansion | +|---|---| +| `join(other, (k, (v1, v2)) -> (k2, v3))` | `join(other) \| map((k, (v1, v2)) -> (k2, v3))`, without building the joined rows | + +[Join fragments](#join-fragments) are sugar for a chain of `join`s. + +#### Join fragments + +A join fragment writes a multiway join as a rule. +Each atom names a collection and its key and value patterns, and `*` joins the atoms left to right: + +``` +let revenue = orders(o, (c, _)) * customer(c, (_, n)) * nation(n, (name, r)) + | map(((o, c, n, name, r), ()) -> (name, o)); +``` + +A fragment's rows are keyed by its variables in order of first appearance, with a unit value; one variable is the key itself, and no variables are `()`. +A repeated variable tests equality, and a literal in a pattern tests that value. +Every later atom's key must use a variable bound by an earlier atom; the join order is the order written, so put selective atoms first. +Fragments are experimental: the parser expands them into plain joins, and they may change. + +### Reductions + +The reference form is `reduce((k, vals) -> out)`, or `reduce(limit n, (k, vals) -> out)`. +It groups rows by key. +`vals` is the key's `List` of `(value, diff)` in value order, the values whose diffs do not sum to zero, and only the first `n` of them with a limit. +`out` is a `List` of `(value, diff)`, the key's result, and a key whose `out` is empty has no row. + +| Sugar | Result per key | Expansion | +|---|---|---| +| `min` | the least value | `reduce(limit 1, (k, vals) -> (vals head .0, 1) enlist)` | +| `count` | the sum of the diffs | `map((k, _) -> (k, ())) \| reduce((k, vals) -> (vals head .1, 1) enlist)` | +| `sum` | the sum of the integer values, each weighted by its diff | `lift \| join_function((k, (v, t)) -> ((k, ()), t, v) enlist) \| count` | +| `distinct` | each present value once | `reduce((k, vals) -> vals map ((v, _) -> (v, 1)))` | +| `collect` | a `List` of the values, each repeated by its count | `reduce((k, vals) -> (vals map ((v, d) -> (v, (0, d) range) cap_list map ((v, _) -> v)) flatten .1, 1) enlist)` | + +When the value is not `()`, `count` drops it first and `distinct` keys by the whole record, so neither reduce reads a key's values together. + +`sum` moves each value into its diff and counts, so it costs what `count` costs. Its `lift` only hands `join_function` each row's own time, which leaves the time as it is at any depth ([`sum`](examples/recipes/sum.ddp)). +A key whose diffs sum to zero has no row, so a total of zero is absent. +There is no `max` or top-k operator; negate the value and use `min` or `reduce(limit n, ...)`, through a tree of groups for large inputs ([`top_k`](examples/recipes/top_k.ddp)). + +### Operators that read time + +These operators are not functions of data alone: their results depend on the times at which rows arrived, and not only on the rows present at each time. +None of them is sugar. + +| Operator | Result | +|---|---| +| `lift` | each value paired with its row's time: `(k, (v, t))` | +| `half_join(other, strict)` | for a lifted stream `(k, (v1, t))`: each update meets `other`'s rows `(k, v2)` whose times come before its own (strictly when `strict`, a bool written `0` or `1`, is `1`), giving `(k, ((v1, v2), t2))` at the update's time, where `t2` is the later of the two times, coordinate by coordinate; `other`'s own changes produce nothing | +| `half_join(other, strict, f)` | `half_join(other, strict) \| map(f)`, without building the joined rows | + +`half_join` is the building block of delta joins, which never build the intermediate results of a multiway join; see the [`triangles`](examples/recipes/triangles.ddp) recipe. +A loop's round is the last coordinate of `lift`'s time. +`lift` with `delay` makes an input row count at its own time only ([`upsert`](examples/recipes/upsert.ddp)). + +### Arrangements and inspection + +| Operator | Result | +|---|---| +| `arrange` | the collection, indexed by key; joins and reductions arrange their inputs themselves, and share an arrangement when they can | +| `inspect(label)` | the collection unchanged, with its updates printed to the server's stdout | + +## Lambdas + +A lambda runs `let pattern = input in body`, where `input` is the tuple of the operator's rows: `(key, value)` for a row-by-row operator, `(key, (left value, right value))` for a join, and `(key, vals)` for `reduce`. +The language is corgi's. + +Patterns are names, `_`, and tuples of patterns, each optionally with a shape: `((src, dst), ())`, `(k, v : int)`. +Operators follow their operands: `(a, b) add`, `xs len`, `(mask, a, b) select`. +Stages chain by juxtaposition, `xs map (x -> (x, 1) add) fold_add`, so nothing needs parentheses for grouping. + +| Form | Meaning | +|---|---| +| `x.0` | a tuple's field | +| `(a, b)`, `(a)`, `()` | a tuple, a one-field tuple, unit | +| `let p = e in body` | bind a value | +| `7`, `-3`, `2.5`, `"text"` | literals; a string is its `List(int)` of bytes | +| `[1, 2]`, `[] : List(int)` | a constant list | +| `xs map (x -> e)` | apply to each element | +| `(seed, xs) fold ((acc, x) -> e)`, `scan` | a fold, and its running form | +| `x inject Variant` | build a sum | +| `s match (A (x -> e1), B (y -> e2))` | take a sum apart | +| `s map_variant A (x -> e)` | change one variant | +| `(x, c) branch Option map_variant None (_ -> ())` | `Some(x)` when `c` is nonzero, else `None` | +| `x call name` | a host function, such as `fsqrt` or `bit_length` | +| `e : shape` | an ascription: `e` has this shape | + +A lambda reads only its own input and literals; to use a list beside another value inside `map`, pair them first with `cap_list`. + +### Scalar operations + +Integers are signed 64-bit; arithmetic takes its kind from its operands, so `(1, 2.5) add` is a shape error. + +| Group | Operations | +|---|---| +| arithmetic | `add sub mul div rem neg min max`; `to_float` | +| comparison | `eq ne lt le gt ge` give `int` 0 or 1, comparing any shapes in structural order, strings included | +| logic | `(a, b) and`, `or`, `xor` bitwise; `x and 255` masks; `(a, b, c) all`, `any` | +| choice | `(mask, then, else) select` | +| 64-bit words | `add_b64 sub_b64 mul_b64`; `shl_b64 k shr_b64 k rotl_b64 k rotr_b64 k` | +| lists | `len enlist append flatten range zip head get gather slices chunk k find cut adjacent filter sort dedup group sort_by sort_limit k transpose` | +| list reductions | `fold_add fold_mul fold_min fold_max fold_all fold_any`, and `scan_` of each | +| capture | `cap_list` pairs a context with every list element; `cap_sum` distributes one into every sum lane | +| checked forms | `try_get try_gather try_zip try_chunk` give a two-lane sum, whose second lane holds the rows the plain form would have lost something on | +| text | `split ","`, `parse_int` | +| other | `hash` (a stable 64-bit content hash), `unit` | +| host functions, by `call` | `fabs fsqrt fexp fln ffloor fceil fround fsin fcos ftan fpow fpowi fint fmin fmax`; `feq fne flt fle fgt fge` (IEEE comparisons); `bit_length` | + +`zip`, `get` and `gather` read the zero of the element's shape past the end of a list, and `chunk` drops a short last piece; the `try_` forms report those cases. + +## Time and scopes + +A program's root time is one integer by default, the time the client gives each feed. +A program loaded with `width=N` has root times of `N` coordinates, written `[x,y]` in commands, compared coordinate by coordinate. +The client advances an input's time with `seal`; a read at time `t` is complete once every input it depends on, fed or bound, is past `t`. + +Inside a nested scope the time gains one coordinate per level, the loop's round: `(epoch, round)` one level down. +A `delay` or `join_function` there names every coordinate, as in `delay((_, d) -> (0, d))`. +`lift` gives the whole time, a tuple even at the root: `(k, (v, (t)))` for a one-coordinate time. + +## Iteration + +A nested scope, `name: { ... }`, computes a fixed point. + +1. Each feedback variable starts empty. +2. Each round, every variable becomes its `set` value, computed from the previous round's values. +3. The scope's result is the value each variable stops changing at. + +From outside, `name::x` is the fixed point of `x`. +A scope can hold negation, reductions and further scopes. +A recursion that never stops changing never finishes, and holds up every program on the server. + +Prioritized iteration uses `delay` to hold some rows back to later rounds, so that work likely to be undone waits ([`connected_components`](examples/recipes/connected_components.ddp)). +It can make the first computation much faster, and some later edits slower. + +At the root, `var` and `set` carry state from one time to the next instead. +A root variable is empty at first, and at each time `t` holds what its `set` held at `t - 1`. +Nothing iterates, and anything may feed the `set`, as the cycle passes through the step ([`upsert`](examples/recipes/upsert.ddp)). +Root variables need a one-coordinate root time. + +## Design positions + +These are deliberate, and differ from what SQL users expect. + +- **Inputs are multisets.** Feeding a row twice gives it a count of 2, and retracting an absent row gives it a count of -1. `distinct` keeps any row whose count is not zero. Replacing a key's value is a program's job ([`upsert`](examples/recipes/upsert.ddp)). +- **Arithmetic does not fail.** Integers wrap: `(9223372036854775807, 1) add` is `-9223372036854775808`. `(x, 0) div` is 0 and `(x, 0) rem` is `x`; division rounds toward zero, so `(-7, 2) div` is -3. A program that needs errors checks its operands and builds a `Result`. +- **There is no NULL.** A missing value is a variant. Declare an enum for it ([`left_join`](examples/recipes/left_join.ddp)); `() inject None` is refused, because `Option`'s `Some` lane has no shape until a value supplies one. +- **Nothing is durable.** A server's programs and data last as long as its process. +- **One server is one trust domain.** Programs share one command queue and one set of workers: a slow program delays the others, and a program that never finishes or crashes a worker stops them all. +- **Inputs belong to their program.** Another program cannot read an input, only an output. Changing a program's schema is a migration; a stable program that only publishes its inputs, read by programs whose inputs are bound to them, keeps views replaceable. +- **There are no dates or times of day.** Use integers, such as days or seconds since an epoch. +- **Joins match keys exactly.** A range or approximate join is a re-keying into buckets first. +- **Strings are byte lists.** A feed and a peek spell `"ann"` as `list(97,110,110)`. + +## Seeing what a program does + +`ddir-plan file.ddp` prints a program's plan and its inferred shapes without running it, and exits non-zero when a program has a problem. +`peek` reads an output at a time, `tail` follows its changes, and `inspect(label)` prints a collection's updates to the server's stdout. +Only outputs can be read, so to see an intermediate collection, publish it as an output. diff --git a/interactive/README.md b/interactive/README.md index 7057a2766..a89614d80 100644 --- a/interactive/README.md +++ b/interactive/README.md @@ -1,6 +1,6 @@ # DDIR -[Server command reference](server/README.md#server-command-reference) +[Language reference](LANGUAGE.md) · [Recipes](examples/recipes/README.md) · [Server command reference](server/README.md#server-command-reference) An interpreted language (and runtime) for computation over continually changing collections. diff --git a/interactive/examples/recipes/README.md b/interactive/examples/recipes/README.md new file mode 100644 index 000000000..eadb765a1 --- /dev/null +++ b/interactive/examples/recipes/README.md @@ -0,0 +1,30 @@ +# Recipes + +Each recipe is a small program for one idiom, with a session that runs it on a few rows. +The idioms are the ones that make DDIR programs fast or correct, and that are hard to find from the operator list alone. + +Run a recipe from `interactive/`, and compare with its recorded transcript: + +```sh +cargo run --release -p ddir-server < examples/recipes/upsert.txt +``` + +`NAME.ddp` is the program and explains the idiom in its comments. +`NAME.txt` feeds it, seals its inputs, and peeks the results. +`NAME.out` is the transcript, checked at one and three workers by `server/tests/recipes.rs`. +After an intended change, `UPDATE_RECIPES=1 cargo test --release -p ddir-server --test recipes` records the transcripts again. + +| Recipe | Idiom | Use it for | +|---|---|---| +| [`sum`](sum.ddp) | `sum` adds each key's values, weighted by their diffs | sums of integers, incrementally | +| [`top_k`](top_k.ddp) | a tree of `reduce(limit k)` groups | top k over many rows, with small updates | +| [`retention`](retention.ddp) | `lift` and `delay` retract each row a fixed time after it arrives | sliding windows, expiry | +| [`upsert`](upsert.ddp) | a root `var` holds the table one time ago; writes count at their own time only | a keyed table with put and delete | +| [`left_join`](left_join.ddp) | the unmatched rows are the input minus the matched ones; a declared `enum` for the missing value | outer joins, without NULL | +| [`connected_components`](connected_components.ddp) | `delay` holds back large labels, so small ones spread first | label propagation, prioritized iteration | +| [`shortest_paths`](shortest_paths.ddp) | `delay` to bucket `d / Δ`, spaced several rounds apart | weighted shortest paths, delta stepping | +| [`triangles`](triangles.ddp) | `lift` and `half_join` along each use of a relation | multiway joins without large intermediate results | + +Two facts the recipes lean on: +- **Inputs are multisets.** A row's count is the sum of its diffs, so a second put of a key adds a second row; replacing a value is a program's job, as in `upsert`. +- **`(e)` is a one-field tuple, not `e`.** Parentheses build a tuple. Write `t` where you mean the value itself; `(t)` is a different shape. diff --git a/interactive/examples/recipes/connected_components.ddp b/interactive/examples/recipes/connected_components.ddp new file mode 100644 index 000000000..5f99425c6 --- /dev/null +++ b/interactive/examples/recipes/connected_components.ddp @@ -0,0 +1,22 @@ +-- Connected components by label propagation, with prioritized iteration. +-- +-- Each node starts labeled by itself, and repeatedly takes the least label among its neighbors. +-- Unprioritized, every node's own label starts spreading in the first round, and most of that +-- work is undone when a smaller label arrives. Here `delay` holds node n's own label back to round +-- 256 * bit_length(n), so small labels spread before large ones enter; `scc.ddp` does the same. +-- On a 1M-edge graph this took the first computation from 1.39 s to 0.55 s. +-- The cost moves to some later edits: deleting and re-adding a bridge to a long path took +-- longer with the delay than without (0.7 s and 3.4 s against 37 ms, for a 300-node path). + +let raw = input 0 : (int, int); -- an undirected edge, either way round +let edges = raw + (raw | map((a, b) -> (b, a))) | distinct; +let nodes = edges | map((a, _) -> (a, ())) | distinct; + +cc: { + -- The inner time is (epoch, round): `0` leaves the epoch alone and sets the round. + let seeds = nodes | map((n, _) -> (n, n)) | delay((_, n) -> (0, (256, n call bit_length) mul)); + let proposals = labels | join(edges, (_, (label, dst)) -> (dst, label)); + set labels = seeds + proposals | min; +} + +output "labels" = cc::labels; diff --git a/interactive/examples/recipes/connected_components.out b/interactive/examples/recipes/connected_components.out new file mode 100644 index 000000000..11d84be08 --- /dev/null +++ b/interactive/examples/recipes/connected_components.out @@ -0,0 +1,27 @@ +_1 ok installed "cc" +_2 ok fed "cc" input 0 at t=0 +_3 ok fed "cc" input 0 at t=0 +_4 ok fed "cc" input 0 at t=0 +_5 ok sealed "cc" input 0 to 1 +_6 data 1 val=1 +_6 data 2 val=1 +_6 data 3 val=1 +_6 data 4 val=4 +_6 data 5 val=4 +_6 ok as-of=0 +_7 ok fed "cc" input 0 at t=1 +_8 ok sealed "cc" input 0 to 2 +_9 data 1 val=1 +_9 data 2 val=1 +_9 data 3 val=1 +_9 data 4 val=1 +_9 data 5 val=1 +_9 ok as-of=1 +_10 ok fed "cc" input 0 at t=2 +_11 ok sealed "cc" input 0 to 3 +_12 data 2 val=2 +_12 data 3 val=2 +_12 data 4 val=2 +_12 data 5 val=2 +_12 ok as-of=2 +_13 ok bye diff --git a/interactive/examples/recipes/connected_components.txt b/interactive/examples/recipes/connected_components.txt new file mode 100644 index 000000000..90ed17f3d --- /dev/null +++ b/interactive/examples/recipes/connected_components.txt @@ -0,0 +1,21 @@ +# Connected components with prioritized label propagation: see connected_components.ddp. +# Run from interactive/: ddir_server < examples/recipes/connected_components.txt +load cc from examples/recipes/connected_components.ddp + +# Time 0: two components, {1, 2, 3} and {4, 5}. +feed cc 0 1 val=2 time=0 +feed cc 0 3 val=2 time=0 +feed cc 0 5 val=4 time=0 +seal cc 0 1 +peek cc::labels + +# Time 1: the edge 3-4 joins them. +feed cc 0 3 val=4 time=1 +seal cc 0 2 +peek cc::labels + +# Time 2: removing 1-2 leaves node 1 with no edges, so no label; the rest take label 2. +feed cc 0 1 val=2 time=2 diff=-1 +seal cc 0 3 +peek cc::labels +exit diff --git a/interactive/examples/recipes/left_join.ddp b/interactive/examples/recipes/left_join.ddp new file mode 100644 index 000000000..92b1e628a --- /dev/null +++ b/interactive/examples/recipes/left_join.ddp @@ -0,0 +1,17 @@ +-- A left join: every order, with its customer's name when there is one. +-- +-- There is no NULL; a missing value is a variant. Declaring the enum gives each variant its +-- payload shape, so `() inject Unknown` needs nothing more. (The built-in `Option` cannot yet +-- build `None` this way: its `Some` lane has no shape until a value supplies one.) +-- The unmatched rows are the input minus the ones the join matched. + +enum Name = Known List(int) | Unknown; + +let orders = input 0 : (int, int); -- (order, customer) +let customers = input 1 : (int, List(int)); -- (customer, name); a string is a list of bytes + +let by_customer = orders | map((order, customer) -> (customer, order)); +let matched = by_customer | join(customers, (c, (order, name)) -> (order, (c, name inject Known))); +let unmatched = by_customer - (by_customer | join(customers, (c, (order, _)) -> (c, order))); + +output "orders" = matched + (unmatched | map((c, order) -> (order, (c, () inject Unknown)))); diff --git a/interactive/examples/recipes/left_join.out b/interactive/examples/recipes/left_join.out new file mode 100644 index 000000000..94d55615e --- /dev/null +++ b/interactive/examples/recipes/left_join.out @@ -0,0 +1,16 @@ +_1 ok installed "lj" +_2 ok fed "lj" input 1 at t=0 +_3 ok fed "lj" input 0 at t=0 +_4 ok fed "lj" input 0 at t=0 +_5 ok sealed "lj" input 0 to 1 +_6 ok sealed "lj" input 1 to 1 +_7 data 100 val=tuple(1,inject(0,list(97,110,110))) +_7 data 101 val=tuple(2,inject(1,tuple())) +_7 ok as-of=0 +_8 ok fed "lj" input 1 at t=1 +_9 ok sealed "lj" input 0 to 2 +_10 ok sealed "lj" input 1 to 2 +_11 data 100 val=tuple(1,inject(0,list(97,110,110))) +_11 data 101 val=tuple(2,inject(0,list(98,111))) +_11 ok as-of=1 +_12 ok bye diff --git a/interactive/examples/recipes/left_join.txt b/interactive/examples/recipes/left_join.txt new file mode 100644 index 000000000..27bab4ffb --- /dev/null +++ b/interactive/examples/recipes/left_join.txt @@ -0,0 +1,19 @@ +# A left join with a declared enum for the missing side: see left_join.ddp. +# Run from interactive/: ddir_server < examples/recipes/left_join.txt +load lj from examples/recipes/left_join.ddp + +# A feed spells a string as its bytes: "ann" is list(97,110,110). +feed lj 1 1 val=list(97,110,110) time=0 +feed lj 0 100 val=1 time=0 +feed lj 0 101 val=2 time=0 +seal lj 0 1 +seal lj 1 1 +# Order 101's customer is unknown: `inject(1,tuple())`. Order 100's is `inject(0,)`. +peek lj::orders + +# Customer 2 arrives, and order 101 moves from Unknown to Known. +feed lj 1 2 val=list(98,111) time=1 +seal lj 0 2 +seal lj 1 2 +peek lj::orders +exit diff --git a/interactive/examples/recipes/retention.ddp b/interactive/examples/recipes/retention.ddp new file mode 100644 index 000000000..636ba58a4 --- /dev/null +++ b/interactive/examples/recipes/retention.ddp @@ -0,0 +1,13 @@ +-- A sliding window: count each key's events over the last 3 times. +-- +-- `lift` pairs each row with its time t, and `delay` moves a copy of the row to t + 3. +-- Subtracting that copy retracts each event 3 times after it arrived, so the window holds only +-- recent events, and its state stays the size of the window rather than of the whole history. +-- Joining against a clock input instead would redo the work for every event at every tick. + +let events = input 0 : (int, ()); -- an event for a key, at the time it is fed + +let expired = events | lift | delay((_, (_, (t))) -> (t, 3) add) | map((k, _) -> (k, ())); +let recent = events - expired; + +output "recent" = recent | count; diff --git a/interactive/examples/recipes/retention.out b/interactive/examples/recipes/retention.out new file mode 100644 index 000000000..c440ea399 --- /dev/null +++ b/interactive/examples/recipes/retention.out @@ -0,0 +1,16 @@ +_1 ok installed "win" +_2 ok fed "win" input 0 at t=0 +_3 ok fed "win" input 0 at t=1 +_4 ok fed "win" input 0 at t=1 +_5 ok fed "win" input 0 at t=2 +_6 ok sealed "win" input 0 to 3 +_7 data 7 val=3 +_7 data 8 val=1 +_7 ok as-of=2 +_8 ok sealed "win" input 0 to 5 +_9 data 7 val=2 +_9 data 8 val=1 +_9 ok as-of=3 +_10 data 7 val=1 +_10 ok as-of=4 +_11 ok bye diff --git a/interactive/examples/recipes/retention.txt b/interactive/examples/recipes/retention.txt new file mode 100644 index 000000000..b3bfdedcf --- /dev/null +++ b/interactive/examples/recipes/retention.txt @@ -0,0 +1,17 @@ +# A sliding window with `delay`: see retention.ddp. +# Run from interactive/: ddir_server < examples/recipes/retention.txt +load win from examples/recipes/retention.ddp + +feed win 0 7 time=0 +feed win 0 7 time=1 +feed win 0 8 time=1 +feed win 0 7 time=2 +seal win 0 3 +# Times 0, 1 and 2 are all in the window at time 2. +peek win::recent + +# At time 3 the event from time 0 leaves the window; at time 4, those from time 1 leave. +seal win 0 5 +peek win::recent as-of 3 +peek win::recent as-of 4 +exit diff --git a/interactive/examples/recipes/shortest_paths.ddp b/interactive/examples/recipes/shortest_paths.ddp new file mode 100644 index 000000000..c2510fc82 --- /dev/null +++ b/interactive/examples/recipes/shortest_paths.ddp @@ -0,0 +1,22 @@ +-- Single-source shortest paths with delta stepping. +-- +-- Plain relaxation in a loop finds every path, but on weighted graphs it keeps improving long +-- tentative distances that a shorter path later replaces. Delta stepping settles distances in +-- buckets of width Δ, nearest first. Here `delay` puts a proposed distance d at round +-- (d / 64) * 16: bucket d / 64, spaced 16 rounds apart, so a bucket's hop-by-hop relaxations +-- finish before the next bucket starts. With the spacing at 1 a bucket's relaxations run into +-- the next bucket's and the delay gains nothing. +-- On a 500 x 500 grid with weights 1 to 100 this took the first computation from 64.5 s to 0.9 s. +-- Choose Δ near the typical edge weight, and the spacing above the hops a bucket needs. + +let edges = input 0 : (int, (int, int)); -- (src, (dst, weight)) +let roots = input 1 : (int, ()); -- the sources, at distance 0 + +sssp: { + let proposals = dist + | join(edges, (_, (d, (dst, w))) -> (dst, (d, w) add)) + | delay((_, d) -> (0, ((d, 64) div, 16) mul)); + set dist = (roots | map((n, ()) -> (n, 0))) + proposals | min; +} + +output "dist" = sssp::dist; diff --git a/interactive/examples/recipes/shortest_paths.out b/interactive/examples/recipes/shortest_paths.out new file mode 100644 index 000000000..246a482f5 --- /dev/null +++ b/interactive/examples/recipes/shortest_paths.out @@ -0,0 +1,23 @@ +_1 ok installed "sp" +_2 ok fed "sp" input 0 at t=0 +_3 ok fed "sp" input 0 at t=0 +_4 ok fed "sp" input 0 at t=0 +_5 ok fed "sp" input 0 at t=0 +_6 ok fed "sp" input 1 at t=0 +_7 ok sealed "sp" input 0 to 1 +_8 ok sealed "sp" input 1 to 1 +_9 data 1 val=0 +_9 data 2 val=200 +_9 data 3 val=400 +_9 data 4 val=410 +_9 ok as-of=0 +_10 ok fed "sp" input 0 at t=1 +_11 ok fed "sp" input 0 at t=1 +_12 ok sealed "sp" input 0 to 2 +_13 ok sealed "sp" input 1 to 2 +_14 data 1 val=0 +_14 data 2 val=200 +_14 data 3 val=500 +_14 data 4 val=510 +_14 ok as-of=1 +_15 ok bye diff --git a/interactive/examples/recipes/shortest_paths.txt b/interactive/examples/recipes/shortest_paths.txt new file mode 100644 index 000000000..2ec40885e --- /dev/null +++ b/interactive/examples/recipes/shortest_paths.txt @@ -0,0 +1,21 @@ +# Shortest paths with delta stepping: see shortest_paths.ddp. +# Run from interactive/: ddir_server < examples/recipes/shortest_paths.txt +load sp from examples/recipes/shortest_paths.ddp + +# Time 0: 1 -> 2 -> 3 costs 200 + 200; the direct edge 1 -> 3 costs 500. +feed sp 0 1 val=tuple(2,200) time=0 +feed sp 0 2 val=tuple(3,200) time=0 +feed sp 0 1 val=tuple(3,500) time=0 +feed sp 0 3 val=tuple(4,10) time=0 +feed sp 1 1 time=0 +seal sp 0 1 +seal sp 1 1 +peek sp::dist + +# Time 1: the edge 2 -> 3 gets dearer, and the direct edge wins. +feed sp 0 2 val=tuple(3,200) time=1 diff=-1 +feed sp 0 2 val=tuple(3,900) time=1 +seal sp 0 2 +seal sp 1 2 +peek sp::dist +exit diff --git a/interactive/examples/recipes/sum.ddp b/interactive/examples/recipes/sum.ddp new file mode 100644 index 000000000..0d1969ca0 --- /dev/null +++ b/interactive/examples/recipes/sum.ddp @@ -0,0 +1,9 @@ +-- Sum an integer per key with `sum`, which adds each key's values weighted by their diffs. +-- +-- `sum` is sugar: it moves each value into its row's diff, and `count` adds up the diffs. +-- The sum is maintained incrementally, where folding a list of values would rebuild it per change. +-- A total of zero has no row, like any record whose diffs cancel; it is bounded by `i64`. + +let sales = input 0 : (int, int); -- (store, amount); a refund is a negative amount + +output "total" = sales | sum; diff --git a/interactive/examples/recipes/sum.out b/interactive/examples/recipes/sum.out new file mode 100644 index 000000000..1b660ce9e --- /dev/null +++ b/interactive/examples/recipes/sum.out @@ -0,0 +1,15 @@ +_1 ok installed "sum" +_2 ok fed "sum" input 0 at t=0 +_3 ok fed "sum" input 0 at t=0 +_4 ok fed "sum" input 0 at t=0 +_5 ok fed "sum" input 0 at t=0 +_6 ok sealed "sum" input 0 to 1 +_7 data 1 val=19 +_7 data 2 val=40 +_7 ok as-of=0 +_8 ok fed "sum" input 0 at t=1 +_9 ok fed "sum" input 0 at t=1 +_10 ok sealed "sum" input 0 to 2 +_11 data 1 val=20 +_11 ok as-of=1 +_12 ok bye diff --git a/interactive/examples/recipes/sum.txt b/interactive/examples/recipes/sum.txt new file mode 100644 index 000000000..c93ddd3f6 --- /dev/null +++ b/interactive/examples/recipes/sum.txt @@ -0,0 +1,18 @@ +# Sum per key with `sum`: see sum.ddp. +# Run from interactive/: ddir_server < examples/recipes/sum.txt +load sum from examples/recipes/sum.ddp + +# Time 0: three sales at store 1, one at store 2. +feed sum 0 1 val=5 time=0 +feed sum 0 1 val=7 time=0 +feed sum 0 1 val=7 time=0 +feed sum 0 2 val=40 time=0 +seal sum 0 1 +peek sum::total + +# Time 1: a refund at store 2 brings it to zero, and its row goes away. +feed sum 0 2 val=-40 time=1 +feed sum 0 1 val=1 time=1 +seal sum 0 2 +peek sum::total +exit diff --git a/interactive/examples/recipes/top_k.ddp b/interactive/examples/recipes/top_k.ddp new file mode 100644 index 000000000..c4589c8e3 --- /dev/null +++ b/interactive/examples/recipes/top_k.ddp @@ -0,0 +1,19 @@ +-- The top 3 scores overall, maintained as scores change. +-- +-- `reduce(limit 3, ...)` hands its lambda only a key's first 3 values in value order, so storing +-- the negated score puts the highest first. One group holding every player would redo that +-- group's work on every update; a tree of groups keeps each update small. Here players fall into +-- 64 buckets, each bucket keeps its own top 3, and the root takes the top 3 of those. +-- An update then touches one bucket and the 64 * 3 candidates at the root. +-- Ties order by player, the second field of the value. + +let scores = input 0 : (int, int); -- (player, score) + +let buckets = scores + | map((player, score) -> ((player, 64) rem, (score neg, player))) + | reduce(limit 3, (_, vals) -> vals); +let top = buckets + | map((_, v) -> ((), v)) + | reduce(limit 3, (_, vals) -> vals); + +output "top" = top | map((_, (negated, player)) -> (player, negated neg)); diff --git a/interactive/examples/recipes/top_k.out b/interactive/examples/recipes/top_k.out new file mode 100644 index 000000000..2222a08a3 --- /dev/null +++ b/interactive/examples/recipes/top_k.out @@ -0,0 +1,21 @@ +_1 ok installed "top" +_2 ok fed "top" input 0 at t=0 +_3 ok fed "top" input 0 at t=0 +_4 ok fed "top" input 0 at t=0 +_5 ok fed "top" input 0 at t=0 +_6 ok fed "top" input 0 at t=0 +_7 ok sealed "top" input 0 to 1 +_8 data 2 val=80 +_8 data 65 val=70 +_8 data 129 val=90 +_8 ok as-of=0 +_9 ok fed "top" input 0 at t=1 +_10 ok fed "top" input 0 at t=1 +_11 ok fed "top" input 0 at t=1 +_12 ok fed "top" input 0 at t=1 +_13 ok sealed "top" input 0 to 2 +_14 data 2 val=80 +_14 data 3 val=95 +_14 data 65 val=70 +_14 ok as-of=1 +_15 ok bye diff --git a/interactive/examples/recipes/top_k.txt b/interactive/examples/recipes/top_k.txt new file mode 100644 index 000000000..ef7b655c7 --- /dev/null +++ b/interactive/examples/recipes/top_k.txt @@ -0,0 +1,22 @@ +# Top k through a tree of `reduce(limit k)`: see top_k.ddp. +# Run from interactive/: ddir_server < examples/recipes/top_k.txt +load top from examples/recipes/top_k.ddp + +# Time 0: five players. +feed top 0 1 val=50 time=0 +feed top 0 2 val=80 time=0 +feed top 0 65 val=70 time=0 +feed top 0 3 val=10 time=0 +feed top 0 129 val=90 time=0 +seal top 0 1 +peek top::top + +# Time 1: player 3 climbs to the top, and player 129 drops out: a score change is a retraction +# of the old row and an insertion of the new one. +feed top 0 3 val=10 time=1 diff=-1 +feed top 0 3 val=95 time=1 +feed top 0 129 val=90 time=1 diff=-1 +feed top 0 129 val=20 time=1 +seal top 0 2 +peek top::top +exit diff --git a/interactive/examples/recipes/triangles.ddp b/interactive/examples/recipes/triangles.ddp new file mode 100644 index 000000000..da1f09204 --- /dev/null +++ b/interactive/examples/recipes/triangles.ddp @@ -0,0 +1,40 @@ +-- Triangle counting as a delta join: no collection of wedges is ever built. +-- +-- Triangles are T(a, b, c) :- E(a, b), E(a, c), E(b, c), with each edge pointed from its larger +-- end to its smaller, and b > c. The plain plan joins E with itself into every wedge (a path of +-- two edges) and arranges them all: 60.8M wedges for a 1M-edge graph. A delta join instead +-- handles each change to E once per use of E in the rule, and joins that change against the +-- other two uses with `half_join`, which reads the arranged edges as of the change's time. +-- For the use numbered i, a change meets later uses at or before its time (strict 0) and earlier +-- uses strictly before it (strict 1), so a change that touches two uses counts once. +-- On the 1M-edge graph a single edge update went from 2.86 ms to 0.25 ms. + +let raw = input 0 : (int, int); -- an undirected edge, either way round +let down = raw + | filter((a, b) -> (a, b) ne) + | map((a, b) -> ((((a, b) gt, a, b) select, ((a, b) gt, b, a) select), ())) + | distinct; +let out = down | map(((a, b), ()) -> (a, b)); -- keyed by the larger end +let rev = down | map(((a, b), ()) -> (b, a)); -- keyed by the smaller end +let pairs = down | arrange; -- keyed by the whole edge + +-- A change to E(a, b): find E(a, c), then check E(b, c). +let p1 = out | lift + | half_join(out, 0, (a, ((b, c), t)) -> ((b, c), (a, t))) + | filter(((b, c), _) -> (b, c) gt) + | half_join(pairs, 0, ((b, c), ((a, ()), t)) -> (a, ((b, c), t))); +-- A change to E(a, c): find E(a, b) strictly earlier, then check E(b, c). +let p2 = out | lift + | half_join(out, 1, (a, ((c, b), t)) -> ((b, c), (a, t))) + | filter(((b, c), _) -> (b, c) gt) + | half_join(pairs, 0, ((b, c), ((a, ()), t)) -> (a, ((b, c), t))); +-- A change to E(b, c): find E(a, b) and then E(a, c), both strictly earlier. +let p3 = down | map(((b, c), ()) -> (b, c)) | lift + | half_join(rev, 1, (b, ((c, a), t)) -> ((a, c), (b, t))) + | half_join(pairs, 1, ((a, c), ((b, ()), t)) -> (a, ((b, c), t))); + +-- Each found triangle carries the time of the change that found it; `delay` puts it there. +let triangles = (p1 + p2 + p3) | delay((_, (_, t)) -> t) | map((a, (bc, _)) -> (a, bc)); + +output "triangles" = triangles; +output "total" = triangles | map(_ -> ((), ())) | count; diff --git a/interactive/examples/recipes/triangles.out b/interactive/examples/recipes/triangles.out new file mode 100644 index 000000000..c10767d9c --- /dev/null +++ b/interactive/examples/recipes/triangles.out @@ -0,0 +1,24 @@ +_1 ok installed "tri" +_2 ok fed "tri" input 0 at t=0 +_3 ok fed "tri" input 0 at t=0 +_4 ok fed "tri" input 0 at t=0 +_5 ok fed "tri" input 0 at t=0 +_6 ok fed "tri" input 0 at t=0 +_7 ok fed "tri" input 0 at t=0 +_8 ok fed "tri" input 0 at t=0 +_9 ok sealed "tri" input 0 to 1 +_10 data tuple() val=4 +_10 ok as-of=0 +_11 data 3 val=tuple(2,1) +_11 data 4 val=tuple(2,1) +_11 data 4 val=tuple(3,1) +_11 data 4 val=tuple(3,2) +_11 ok as-of=0 +_12 ok fed "tri" input 0 at t=1 +_13 ok sealed "tri" input 0 to 2 +_14 data tuple() val=2 +_14 ok as-of=1 +_15 data 4 val=tuple(3,1) +_15 data 4 val=tuple(3,2) +_15 ok as-of=1 +_16 ok bye diff --git a/interactive/examples/recipes/triangles.txt b/interactive/examples/recipes/triangles.txt new file mode 100644 index 000000000..85f78d716 --- /dev/null +++ b/interactive/examples/recipes/triangles.txt @@ -0,0 +1,22 @@ +# Triangles as a delta join: see triangles.ddp. +# Run from interactive/: ddir_server < examples/recipes/triangles.txt +load tri from examples/recipes/triangles.ddp + +# Time 0: all six edges among 1, 2, 3, 4 (four triangles), and the edge 4-5 (none). +feed tri 0 1 val=2 time=0 +feed tri 0 1 val=3 time=0 +feed tri 0 1 val=4 time=0 +feed tri 0 2 val=3 time=0 +feed tri 0 2 val=4 time=0 +feed tri 0 3 val=4 time=0 +feed tri 0 4 val=5 time=0 +seal tri 0 1 +peek tri::total +peek tri::triangles + +# Time 1: removing 1-2 leaves the two triangles through 3-4. +feed tri 0 1 val=2 time=1 diff=-1 +seal tri 0 2 +peek tri::total +peek tri::triangles +exit diff --git a/interactive/examples/recipes/upsert.ddp b/interactive/examples/recipes/upsert.ddp new file mode 100644 index 000000000..f400974cc --- /dev/null +++ b/interactive/examples/recipes/upsert.ddp @@ -0,0 +1,25 @@ +-- Upsert: a table keyed by `int`, where a put replaces the key's value and a delete removes it. +-- +-- Inputs are multisets, so feeding `5 val=2` after `5 val=1` leaves both rows. +-- Here the table is instead a root feedback variable: at each time it holds what `now` held +-- one time earlier, and `now` is that table with this time's writes applied. +-- The client never retracts anything; it only feeds the writes. + +let puts = input 0 : (int, int); -- (key, value): set key to value +let dels = input 1 : (int, ()); -- key: remove key + +-- A write counts at its own time only: `lift` pairs each row with its time t, and `delay` +-- moves a copy to t + 1, where subtracting it retracts the row. +let put_now = puts - (puts | lift | delay((_, (_, (t))) -> (t, 1) add) | map((k, (v, _)) -> (k, v))); +let del_now = dels - (dels | lift | delay((_, (_, (t))) -> (t, 1) add) | map((k, _) -> (k, ()))); + +-- Several puts to one key at one time: the least value wins. A put beats a delete at one time. +let written = put_now | min; +let touched = (written | map((k, _) -> (k, ()))) + del_now | distinct; + +var table : (int, int); +let kept = table - (table | join(touched, (k, (v, ())) -> (k, v))); +let now = kept + written; +set table = now; + +output "table" = now; diff --git a/interactive/examples/recipes/upsert.out b/interactive/examples/recipes/upsert.out new file mode 100644 index 000000000..e5343b8af --- /dev/null +++ b/interactive/examples/recipes/upsert.out @@ -0,0 +1,33 @@ +_1 ok installed "upsert" +_2 ok fed "upsert" input 0 at t=0 +_3 ok fed "upsert" input 0 at t=0 +_4 ok sealed "upsert" input 0 to 1 +_5 ok sealed "upsert" input 1 to 1 +_6 data 1 val=10 +_6 data 2 val=20 +_6 ok as-of=0 +_7 ok fed "upsert" input 0 at t=1 +_8 ok fed "upsert" input 0 at t=1 +_9 ok sealed "upsert" input 0 to 2 +_10 ok sealed "upsert" input 1 to 2 +_11 data 1 val=11 +_11 data 2 val=20 +_11 data 3 val=30 +_11 ok as-of=1 +_12 ok fed "upsert" input 1 at t=2 +_13 ok fed "upsert" input 0 at t=2 +_14 ok fed "upsert" input 0 at t=2 +_15 ok sealed "upsert" input 0 to 3 +_16 ok sealed "upsert" input 1 to 3 +_17 data 1 val=11 +_17 data 3 val=31 +_17 ok as-of=2 +_18 ok sealed "upsert" input 0 to 4 +_19 ok sealed "upsert" input 1 to 4 +_20 data 1 val=11 +_20 data 3 val=31 +_20 ok as-of=3 +_21 data 1 val=10 +_21 data 2 val=20 +_21 ok as-of=0 +_22 ok bye diff --git a/interactive/examples/recipes/upsert.txt b/interactive/examples/recipes/upsert.txt new file mode 100644 index 000000000..6b06dfb60 --- /dev/null +++ b/interactive/examples/recipes/upsert.txt @@ -0,0 +1,34 @@ +# Upsert with a root feedback variable: see upsert.ddp. +# Run from interactive/: ddir_server < examples/recipes/upsert.txt +load upsert from examples/recipes/upsert.ddp + +# Time 0: put keys 1 and 2. +feed upsert 0 1 val=10 time=0 +feed upsert 0 2 val=20 time=0 +seal upsert 0 1 +seal upsert 1 1 +peek upsert::table + +# Time 1: overwrite key 1, and add key 3. +feed upsert 0 1 val=11 time=1 +feed upsert 0 3 val=30 time=1 +seal upsert 0 2 +seal upsert 1 2 +peek upsert::table + +# Time 2: delete key 2, and put key 3 twice: the least value wins. +feed upsert 1 2 time=2 +feed upsert 0 3 val=33 time=2 +feed upsert 0 3 val=31 time=2 +seal upsert 0 3 +seal upsert 1 3 +peek upsert::table + +# Time 3: no writes; the table stays as it was. +seal upsert 0 4 +seal upsert 1 4 +peek upsert::table + +# Reading back in time still works. +peek upsert::table as-of 0 +exit diff --git a/interactive/server/tests/recipes.rs b/interactive/server/tests/recipes.rs new file mode 100644 index 000000000..13ec852f9 --- /dev/null +++ b/interactive/server/tests/recipes.rs @@ -0,0 +1,66 @@ +//! The recipes in `examples/recipes`: each `NAME.txt` session, piped into a server's stdin from +//! `interactive/`, must print its recorded transcript `NAME.out`, at one worker and at three. +//! `UPDATE_RECIPES=1` records the one-worker transcripts instead. + +use std::io::Read; +use std::path::{Path, PathBuf}; +use std::process::{Command, Stdio}; +use std::time::{Duration, Instant}; + +fn interactive() -> PathBuf { Path::new(env!("CARGO_MANIFEST_DIR")).join("..") } + +/// The server's stdout for one session. An install's operator count is left out, so a change +/// to the optimizer does not change a recipe's transcript. +fn transcript(session: &Path, workers: usize) -> String { + let mut child = Command::new(env!("CARGO_BIN_EXE_ddir_server")) + .current_dir(interactive()) + .env("DDIR_WORKERS", workers.to_string()) + .env("DDIR_BIND", "127.0.0.1:0") + .env("DDIR_WS_BIND", "127.0.0.1:0") + .stdin(std::fs::File::open(session).unwrap()) + .stdout(Stdio::piped()) + .stderr(Stdio::null()) + .spawn() + .unwrap(); + let deadline = Instant::now() + Duration::from_secs(60); + while child.try_wait().unwrap().is_none() { + if Instant::now() > deadline { + let _ = child.kill(); + panic!("{}: the server did not exit", session.display()); + } + std::thread::sleep(Duration::from_millis(10)); + } + let mut out = String::new(); + child.stdout.take().unwrap().read_to_string(&mut out).unwrap(); + out.lines() + .map(|line| match line.find(" ok installed ") { + Some(_) => line.rsplit_once(" (").map_or(line, |(head, _)| head).to_string(), + None => line.to_string(), + }) + .collect::>() + .join("\n") + "\n" +} + +#[test] +fn recipes_print_their_recorded_transcripts() { + let dir = interactive().join("examples/recipes"); + let mut sessions: Vec = std::fs::read_dir(&dir).unwrap() + .map(|entry| entry.unwrap().path()) + .filter(|path| path.extension().is_some_and(|e| e == "txt")) + .collect(); + sessions.sort(); + assert!(!sessions.is_empty(), "no recipes in {}", dir.display()); + let update = std::env::var("UPDATE_RECIPES").as_deref() == Ok("1"); + for session in &sessions { + let recorded = session.with_extension("out"); + let got = transcript(session, 1); + if update { + std::fs::write(&recorded, &got).unwrap(); + } else { + let want = std::fs::read_to_string(&recorded) + .unwrap_or_else(|e| panic!("{}: {e}", recorded.display())); + assert_eq!(got, want, "{} disagrees with {}", session.display(), recorded.display()); + } + assert_eq!(transcript(session, 3), got, "{} at three workers", session.display()); + } +} diff --git a/interactive/src/parse/pipe.rs b/interactive/src/parse/pipe.rs index da07da3af..7d43b3368 100644 --- a/interactive/src/parse/pipe.rs +++ b/interactive/src/parse/pipe.rs @@ -87,8 +87,8 @@ //! Reusing a variable tests equality. Plain filters run as soon as their fields are bound, //! before the next rekey; a direct first join precedes its first atom's tests. Variants bind //! through typed payload lists in `flatmap`, which can also emit the next lookup key. -//! Rows contain each variable once, in first-appearance order, keyed by their tuple (`(x)` for -//! one variable, `()` for none), with unit value. A test-free final layout composes into +//! Rows contain each variable once, in first-appearance order, keyed by their tuple (the +//! variable itself for one, `()` for none), with unit value. A test-free final layout composes into //! the last join's projection. Weights multiply; nothing implies distinct. //! Even the parsed view shows the expansion. //!