Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
299 changes: 299 additions & 0 deletions interactive/LANGUAGE.md

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion interactive/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# DDIR

[Server command reference](server/README.md#server-command-reference)
[Language reference](LANGUAGE.md) · [Recipes](examples/recipes/README.md) · [Server command reference](server/README.md#server-command-reference)


An interpreted language (and runtime) for computation over continually changing collections.
Expand Down
30 changes: 30 additions & 0 deletions interactive/examples/recipes/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# Recipes

Each recipe is a small program for one idiom, with a session that runs it on a few rows.
The idioms are the ones that make DDIR programs fast or correct, and that are hard to find from the operator list alone.

Run a recipe from `interactive/`, and compare with its recorded transcript:

```sh
cargo run --release -p ddir-server < examples/recipes/upsert.txt
```

`NAME.ddp` is the program and explains the idiom in its comments.
`NAME.txt` feeds it, seals its inputs, and peeks the results.
`NAME.out` is the transcript, checked at one and three workers by `server/tests/recipes.rs`.
After an intended change, `UPDATE_RECIPES=1 cargo test --release -p ddir-server --test recipes` records the transcripts again.

| Recipe | Idiom | Use it for |
|---|---|---|
| [`sum`](sum.ddp) | `sum` adds each key's values, weighted by their diffs | sums of integers, incrementally |
| [`top_k`](top_k.ddp) | a tree of `reduce(limit k)` groups | top k over many rows, with small updates |
| [`retention`](retention.ddp) | `lift` and `delay` retract each row a fixed time after it arrives | sliding windows, expiry |
| [`upsert`](upsert.ddp) | a root `var` holds the table one time ago; writes count at their own time only | a keyed table with put and delete |
| [`left_join`](left_join.ddp) | the unmatched rows are the input minus the matched ones; a declared `enum` for the missing value | outer joins, without NULL |
| [`connected_components`](connected_components.ddp) | `delay` holds back large labels, so small ones spread first | label propagation, prioritized iteration |
| [`shortest_paths`](shortest_paths.ddp) | `delay` to bucket `d / Δ`, spaced several rounds apart | weighted shortest paths, delta stepping |
| [`triangles`](triangles.ddp) | `lift` and `half_join` along each use of a relation | multiway joins without large intermediate results |

Two facts the recipes lean on:
- **Inputs are multisets.** A row's count is the sum of its diffs, so a second put of a key adds a second row; replacing a value is a program's job, as in `upsert`.
- **`(e)` is a one-field tuple, not `e`.** Parentheses build a tuple. Write `t` where you mean the value itself; `(t)` is a different shape.
22 changes: 22 additions & 0 deletions interactive/examples/recipes/connected_components.ddp
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
-- Connected components by label propagation, with prioritized iteration.
--
-- Each node starts labeled by itself, and repeatedly takes the least label among its neighbors.
-- Unprioritized, every node's own label starts spreading in the first round, and most of that
-- work is undone when a smaller label arrives. Here `delay` holds node n's own label back to round
-- 256 * bit_length(n), so small labels spread before large ones enter; `scc.ddp` does the same.
-- On a 1M-edge graph this took the first computation from 1.39 s to 0.55 s.
-- The cost moves to some later edits: deleting and re-adding a bridge to a long path took
-- longer with the delay than without (0.7 s and 3.4 s against 37 ms, for a 300-node path).

let raw = input 0 : (int, int); -- an undirected edge, either way round
let edges = raw + (raw | map((a, b) -> (b, a))) | distinct;
let nodes = edges | map((a, _) -> (a, ())) | distinct;

cc: {
-- The inner time is (epoch, round): `0` leaves the epoch alone and sets the round.
let seeds = nodes | map((n, _) -> (n, n)) | delay((_, n) -> (0, (256, n call bit_length) mul));
let proposals = labels | join(edges, (_, (label, dst)) -> (dst, label));
set labels = seeds + proposals | min;
}

output "labels" = cc::labels;
27 changes: 27 additions & 0 deletions interactive/examples/recipes/connected_components.out
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
_1 ok installed "cc"
_2 ok fed "cc" input 0 at t=0
_3 ok fed "cc" input 0 at t=0
_4 ok fed "cc" input 0 at t=0
_5 ok sealed "cc" input 0 to 1
_6 data 1 val=1
_6 data 2 val=1
_6 data 3 val=1
_6 data 4 val=4
_6 data 5 val=4
_6 ok as-of=0
_7 ok fed "cc" input 0 at t=1
_8 ok sealed "cc" input 0 to 2
_9 data 1 val=1
_9 data 2 val=1
_9 data 3 val=1
_9 data 4 val=1
_9 data 5 val=1
_9 ok as-of=1
_10 ok fed "cc" input 0 at t=2
_11 ok sealed "cc" input 0 to 3
_12 data 2 val=2
_12 data 3 val=2
_12 data 4 val=2
_12 data 5 val=2
_12 ok as-of=2
_13 ok bye
21 changes: 21 additions & 0 deletions interactive/examples/recipes/connected_components.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Connected components with prioritized label propagation: see connected_components.ddp.
# Run from interactive/: ddir_server < examples/recipes/connected_components.txt
load cc from examples/recipes/connected_components.ddp

# Time 0: two components, {1, 2, 3} and {4, 5}.
feed cc 0 1 val=2 time=0
feed cc 0 3 val=2 time=0
feed cc 0 5 val=4 time=0
seal cc 0 1
peek cc::labels

# Time 1: the edge 3-4 joins them.
feed cc 0 3 val=4 time=1
seal cc 0 2
peek cc::labels

# Time 2: removing 1-2 leaves node 1 with no edges, so no label; the rest take label 2.
feed cc 0 1 val=2 time=2 diff=-1
seal cc 0 3
peek cc::labels
exit
17 changes: 17 additions & 0 deletions interactive/examples/recipes/left_join.ddp
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
-- A left join: every order, with its customer's name when there is one.
--
-- There is no NULL; a missing value is a variant. Declaring the enum gives each variant its
-- payload shape, so `() inject Unknown` needs nothing more. (The built-in `Option` cannot yet
-- build `None` this way: its `Some` lane has no shape until a value supplies one.)
-- The unmatched rows are the input minus the ones the join matched.

enum Name = Known List(int) | Unknown;

let orders = input 0 : (int, int); -- (order, customer)
let customers = input 1 : (int, List(int)); -- (customer, name); a string is a list of bytes

let by_customer = orders | map((order, customer) -> (customer, order));
let matched = by_customer | join(customers, (c, (order, name)) -> (order, (c, name inject Known)));
let unmatched = by_customer - (by_customer | join(customers, (c, (order, _)) -> (c, order)));

output "orders" = matched + (unmatched | map((c, order) -> (order, (c, () inject Unknown))));
16 changes: 16 additions & 0 deletions interactive/examples/recipes/left_join.out
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
_1 ok installed "lj"
_2 ok fed "lj" input 1 at t=0
_3 ok fed "lj" input 0 at t=0
_4 ok fed "lj" input 0 at t=0
_5 ok sealed "lj" input 0 to 1
_6 ok sealed "lj" input 1 to 1
_7 data 100 val=tuple(1,inject(0,list(97,110,110)))
_7 data 101 val=tuple(2,inject(1,tuple()))
_7 ok as-of=0
_8 ok fed "lj" input 1 at t=1
_9 ok sealed "lj" input 0 to 2
_10 ok sealed "lj" input 1 to 2
_11 data 100 val=tuple(1,inject(0,list(97,110,110)))
_11 data 101 val=tuple(2,inject(0,list(98,111)))
_11 ok as-of=1
_12 ok bye
19 changes: 19 additions & 0 deletions interactive/examples/recipes/left_join.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# A left join with a declared enum for the missing side: see left_join.ddp.
# Run from interactive/: ddir_server < examples/recipes/left_join.txt
load lj from examples/recipes/left_join.ddp

# A feed spells a string as its bytes: "ann" is list(97,110,110).
feed lj 1 1 val=list(97,110,110) time=0
feed lj 0 100 val=1 time=0
feed lj 0 101 val=2 time=0
seal lj 0 1
seal lj 1 1
# Order 101's customer is unknown: `inject(1,tuple())`. Order 100's is `inject(0,<name>)`.
peek lj::orders

# Customer 2 arrives, and order 101 moves from Unknown to Known.
feed lj 1 2 val=list(98,111) time=1
seal lj 0 2
seal lj 1 2
peek lj::orders
exit
13 changes: 13 additions & 0 deletions interactive/examples/recipes/retention.ddp
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
-- A sliding window: count each key's events over the last 3 times.
--
-- `lift` pairs each row with its time t, and `delay` moves a copy of the row to t + 3.
-- Subtracting that copy retracts each event 3 times after it arrived, so the window holds only
-- recent events, and its state stays the size of the window rather than of the whole history.
-- Joining against a clock input instead would redo the work for every event at every tick.

let events = input 0 : (int, ()); -- an event for a key, at the time it is fed

let expired = events | lift | delay((_, (_, (t))) -> (t, 3) add) | map((k, _) -> (k, ()));
let recent = events - expired;

output "recent" = recent | count;
16 changes: 16 additions & 0 deletions interactive/examples/recipes/retention.out
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
_1 ok installed "win"
_2 ok fed "win" input 0 at t=0
_3 ok fed "win" input 0 at t=1
_4 ok fed "win" input 0 at t=1
_5 ok fed "win" input 0 at t=2
_6 ok sealed "win" input 0 to 3
_7 data 7 val=3
_7 data 8 val=1
_7 ok as-of=2
_8 ok sealed "win" input 0 to 5
_9 data 7 val=2
_9 data 8 val=1
_9 ok as-of=3
_10 data 7 val=1
_10 ok as-of=4
_11 ok bye
17 changes: 17 additions & 0 deletions interactive/examples/recipes/retention.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# A sliding window with `delay`: see retention.ddp.
# Run from interactive/: ddir_server < examples/recipes/retention.txt
load win from examples/recipes/retention.ddp

feed win 0 7 time=0
feed win 0 7 time=1
feed win 0 8 time=1
feed win 0 7 time=2
seal win 0 3
# Times 0, 1 and 2 are all in the window at time 2.
peek win::recent

# At time 3 the event from time 0 leaves the window; at time 4, those from time 1 leave.
seal win 0 5
peek win::recent as-of 3
peek win::recent as-of 4
exit
22 changes: 22 additions & 0 deletions interactive/examples/recipes/shortest_paths.ddp
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
-- Single-source shortest paths with delta stepping.
--
-- Plain relaxation in a loop finds every path, but on weighted graphs it keeps improving long
-- tentative distances that a shorter path later replaces. Delta stepping settles distances in
-- buckets of width Δ, nearest first. Here `delay` puts a proposed distance d at round
-- (d / 64) * 16: bucket d / 64, spaced 16 rounds apart, so a bucket's hop-by-hop relaxations
-- finish before the next bucket starts. With the spacing at 1 a bucket's relaxations run into
-- the next bucket's and the delay gains nothing.
-- On a 500 x 500 grid with weights 1 to 100 this took the first computation from 64.5 s to 0.9 s.
-- Choose Δ near the typical edge weight, and the spacing above the hops a bucket needs.

let edges = input 0 : (int, (int, int)); -- (src, (dst, weight))
let roots = input 1 : (int, ()); -- the sources, at distance 0

sssp: {
let proposals = dist
| join(edges, (_, (d, (dst, w))) -> (dst, (d, w) add))
| delay((_, d) -> (0, ((d, 64) div, 16) mul));
set dist = (roots | map((n, ()) -> (n, 0))) + proposals | min;
}

output "dist" = sssp::dist;
23 changes: 23 additions & 0 deletions interactive/examples/recipes/shortest_paths.out
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
_1 ok installed "sp"
_2 ok fed "sp" input 0 at t=0
_3 ok fed "sp" input 0 at t=0
_4 ok fed "sp" input 0 at t=0
_5 ok fed "sp" input 0 at t=0
_6 ok fed "sp" input 1 at t=0
_7 ok sealed "sp" input 0 to 1
_8 ok sealed "sp" input 1 to 1
_9 data 1 val=0
_9 data 2 val=200
_9 data 3 val=400
_9 data 4 val=410
_9 ok as-of=0
_10 ok fed "sp" input 0 at t=1
_11 ok fed "sp" input 0 at t=1
_12 ok sealed "sp" input 0 to 2
_13 ok sealed "sp" input 1 to 2
_14 data 1 val=0
_14 data 2 val=200
_14 data 3 val=500
_14 data 4 val=510
_14 ok as-of=1
_15 ok bye
21 changes: 21 additions & 0 deletions interactive/examples/recipes/shortest_paths.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Shortest paths with delta stepping: see shortest_paths.ddp.
# Run from interactive/: ddir_server < examples/recipes/shortest_paths.txt
load sp from examples/recipes/shortest_paths.ddp

# Time 0: 1 -> 2 -> 3 costs 200 + 200; the direct edge 1 -> 3 costs 500.
feed sp 0 1 val=tuple(2,200) time=0
feed sp 0 2 val=tuple(3,200) time=0
feed sp 0 1 val=tuple(3,500) time=0
feed sp 0 3 val=tuple(4,10) time=0
feed sp 1 1 time=0
seal sp 0 1
seal sp 1 1
peek sp::dist

# Time 1: the edge 2 -> 3 gets dearer, and the direct edge wins.
feed sp 0 2 val=tuple(3,200) time=1 diff=-1
feed sp 0 2 val=tuple(3,900) time=1
seal sp 0 2
seal sp 1 2
peek sp::dist
exit
9 changes: 9 additions & 0 deletions interactive/examples/recipes/sum.ddp
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
-- Sum an integer per key with `sum`, which adds each key's values weighted by their diffs.
--
-- `sum` is sugar: it moves each value into its row's diff, and `count` adds up the diffs.
-- The sum is maintained incrementally, where folding a list of values would rebuild it per change.
-- A total of zero has no row, like any record whose diffs cancel; it is bounded by `i64`.

let sales = input 0 : (int, int); -- (store, amount); a refund is a negative amount

output "total" = sales | sum;
15 changes: 15 additions & 0 deletions interactive/examples/recipes/sum.out
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
_1 ok installed "sum"
_2 ok fed "sum" input 0 at t=0
_3 ok fed "sum" input 0 at t=0
_4 ok fed "sum" input 0 at t=0
_5 ok fed "sum" input 0 at t=0
_6 ok sealed "sum" input 0 to 1
_7 data 1 val=19
_7 data 2 val=40
_7 ok as-of=0
_8 ok fed "sum" input 0 at t=1
_9 ok fed "sum" input 0 at t=1
_10 ok sealed "sum" input 0 to 2
_11 data 1 val=20
_11 ok as-of=1
_12 ok bye
18 changes: 18 additions & 0 deletions interactive/examples/recipes/sum.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# Sum per key with `sum`: see sum.ddp.
# Run from interactive/: ddir_server < examples/recipes/sum.txt
load sum from examples/recipes/sum.ddp

# Time 0: three sales at store 1, one at store 2.
feed sum 0 1 val=5 time=0
feed sum 0 1 val=7 time=0
feed sum 0 1 val=7 time=0
feed sum 0 2 val=40 time=0
seal sum 0 1
peek sum::total

# Time 1: a refund at store 2 brings it to zero, and its row goes away.
feed sum 0 2 val=-40 time=1
feed sum 0 1 val=1 time=1
seal sum 0 2
peek sum::total
exit
19 changes: 19 additions & 0 deletions interactive/examples/recipes/top_k.ddp
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
-- The top 3 scores overall, maintained as scores change.
--
-- `reduce(limit 3, ...)` hands its lambda only a key's first 3 values in value order, so storing
-- the negated score puts the highest first. One group holding every player would redo that
-- group's work on every update; a tree of groups keeps each update small. Here players fall into
-- 64 buckets, each bucket keeps its own top 3, and the root takes the top 3 of those.
-- An update then touches one bucket and the 64 * 3 candidates at the root.
-- Ties order by player, the second field of the value.

let scores = input 0 : (int, int); -- (player, score)

let buckets = scores
| map((player, score) -> ((player, 64) rem, (score neg, player)))
| reduce(limit 3, (_, vals) -> vals);
let top = buckets
| map((_, v) -> ((), v))
| reduce(limit 3, (_, vals) -> vals);

output "top" = top | map((_, (negated, player)) -> (player, negated neg));
21 changes: 21 additions & 0 deletions interactive/examples/recipes/top_k.out
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
_1 ok installed "top"
_2 ok fed "top" input 0 at t=0
_3 ok fed "top" input 0 at t=0
_4 ok fed "top" input 0 at t=0
_5 ok fed "top" input 0 at t=0
_6 ok fed "top" input 0 at t=0
_7 ok sealed "top" input 0 to 1
_8 data 2 val=80
_8 data 65 val=70
_8 data 129 val=90
_8 ok as-of=0
_9 ok fed "top" input 0 at t=1
_10 ok fed "top" input 0 at t=1
_11 ok fed "top" input 0 at t=1
_12 ok fed "top" input 0 at t=1
_13 ok sealed "top" input 0 to 2
_14 data 2 val=80
_14 data 3 val=95
_14 data 65 val=70
_14 ok as-of=1
_15 ok bye
22 changes: 22 additions & 0 deletions interactive/examples/recipes/top_k.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Top k through a tree of `reduce(limit k)`: see top_k.ddp.
# Run from interactive/: ddir_server < examples/recipes/top_k.txt
load top from examples/recipes/top_k.ddp

# Time 0: five players.
feed top 0 1 val=50 time=0
feed top 0 2 val=80 time=0
feed top 0 65 val=70 time=0
feed top 0 3 val=10 time=0
feed top 0 129 val=90 time=0
seal top 0 1
peek top::top

# Time 1: player 3 climbs to the top, and player 129 drops out: a score change is a retraction
# of the old row and an insertion of the new one.
feed top 0 3 val=10 time=1 diff=-1
feed top 0 3 val=95 time=1
feed top 0 129 val=90 time=1 diff=-1
feed top 0 129 val=20 time=1
seal top 0 2
peek top::top
exit
Loading
Loading