Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 16 additions & 16 deletions interactive/LANGUAGE.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

DDIR programs describe collections that change over time, and the server keeps every collection up to date as its inputs change.
This page is the reference: what a program is made of, every collection operator, the scalar language inside operators, time, iteration, and the positions the language takes on purpose.
For worked examples of the idioms that make programs fast or correct, see the [recipes](examples/recipes/README.md).
For programs at work in the server, see the [examples](examples/README.md).
For the commands that load programs, feed them and read them, see the [server reference](server/README.md#server-command-reference).

## A first program
Expand All @@ -17,15 +17,15 @@ output "total" = sales | sum;
A session loads it, feeds rows at times, closes those times, and reads the result:

```
load sum from examples/recipes/sum.ddp
feed sum 0 1 val=5 time=0
feed sum 0 2 val=40 time=0
seal sum 0 1
peek sum::total
load sales from sales.ddp
feed sales 0 1 val=5 time=0
feed sales 0 2 val=40 time=0
seal sales 0 1
peek sales::total
```

Later feeds at later times update `total`; nothing is recomputed from scratch.
This program is the [`sum`](examples/recipes/sum.ddp) recipe.
[`examples/access.txt`](examples/access.txt) is a whole session.

## Collections

Expand Down Expand Up @@ -124,7 +124,7 @@ The `time` is in the shape of the scope's time (see [Time](#time-and-scopes)), a
A `filter` keeps the rows where `mask` is a nonzero `int`.
Inside a nested scope each `0` above is a time of every coordinate, such as `(0, 0)` one level down.

`delay` retracts rows after a fixed time ([`retention`](examples/recipes/retention.ddp)) and orders work within a loop ([`connected_components`](examples/recipes/connected_components.ddp), [`shortest_paths`](examples/recipes/shortest_paths.ddp)).
`delay` retracts rows after a fixed time ([`retention`](examples/retention.ddp)) and orders work within a loop ([`components`](examples/components.ddp), [`shortest_paths`](examples/shortest_paths.ddp)).

`a + b` is the union of two collections, adding diffs.
It is its own operator, as it reads two inputs, and `a - b` is sugar for `a + (b | negate)`.
Expand Down Expand Up @@ -174,9 +174,9 @@ It groups rows by key.

When the value is not `()`, `count` drops it first and `distinct` keys by the whole record, so neither reduce reads a key's values together.

`sum` moves each value into its diff and counts, so it costs what `count` costs. Its `lift` only hands `join_function` each row's own time, which leaves the time as it is at any depth ([`sum`](examples/recipes/sum.ddp)).
`sum` moves each value into its diff and counts, so it costs what `count` costs. Its `lift` only hands `join_function` each row's own time, which leaves the time as it is at any depth.
A key whose diffs sum to zero has no row, so a total of zero is absent.
There is no `max` or top-k operator; negate the value and use `min` or `reduce(limit n, ...)`, through a tree of groups for large inputs ([`top_k`](examples/recipes/top_k.ddp)).
There is no `max` or top-k operator; negate the value and use `min` or `reduce(limit n, ...)`, through a tree of groups for large inputs ([`top_k`](examples/top_k.ddp)).

### Operators that read time

Expand All @@ -189,9 +189,9 @@ None of them is sugar.
| `half_join(other, strict)` | for a lifted stream `(k, (v1, t))`: each update meets `other`'s rows `(k, v2)` whose times come before its own (strictly when `strict`, a bool written `0` or `1`, is `1`), giving `(k, ((v1, v2), t2))` at the update's time, where `t2` is the later of the two times, coordinate by coordinate; `other`'s own changes produce nothing |
| `half_join(other, strict, f)` | `half_join(other, strict) \| map(f)`, without building the joined rows |

`half_join` is the building block of delta joins, which never build the intermediate results of a multiway join; see the [`triangles`](examples/recipes/triangles.ddp) recipe.
`half_join` is the building block of delta joins, which never build the intermediate results of a multiway join; see the [`triangles`](examples/triangles.ddp) example.
A loop's round is the last coordinate of `lift`'s time.
`lift` with `delay` makes an input row count at its own time only ([`upsert`](examples/recipes/upsert.ddp)).
`lift` with `delay` makes an input row count at its own time only ([`upsert`](examples/upsert.ddp)).

### Arrangements and inspection

Expand Down Expand Up @@ -270,21 +270,21 @@ From outside, `name::x` is the fixed point of `x`.
A scope can hold negation, reductions and further scopes.
A recursion that never stops changing never finishes, and holds up every program on the server.

Prioritized iteration uses `delay` to hold some rows back to later rounds, so that work likely to be undone waits ([`connected_components`](examples/recipes/connected_components.ddp)).
Prioritized iteration uses `delay` to hold some rows back to later rounds, so that work likely to be undone waits ([`components`](examples/components.ddp)).
It can make the first computation much faster, and some later edits slower.

At the root, `var` and `set` carry state from one time to the next instead.
A root variable is empty at first, and at each time `t` holds what its `set` held at `t - 1`.
Nothing iterates, and anything may feed the `set`, as the cycle passes through the step ([`upsert`](examples/recipes/upsert.ddp)).
Nothing iterates, and anything may feed the `set`, as the cycle passes through the step ([`upsert`](examples/upsert.ddp)).
Root variables need a one-coordinate root time.

## Design positions

These are deliberate, and differ from what SQL users expect.

- **Inputs are multisets.** Feeding a row twice gives it a count of 2, and retracting an absent row gives it a count of -1. `distinct` keeps any row whose count is not zero. Replacing a key's value is a program's job ([`upsert`](examples/recipes/upsert.ddp)).
- **Inputs are multisets.** Feeding a row twice gives it a count of 2, and retracting an absent row gives it a count of -1. `distinct` keeps any row whose count is not zero. Replacing a key's value is a program's job ([`upsert`](examples/upsert.ddp)).
- **Arithmetic does not fail.** Integers wrap: `(9223372036854775807, 1) add` is `-9223372036854775808`. `(x, 0) div` is 0 and `(x, 0) rem` is `x`; division rounds toward zero, so `(-7, 2) div` is -3. A program that needs errors checks its operands and builds a `Result`.
- **There is no NULL.** A missing value is a variant. Declare an enum for it ([`left_join`](examples/recipes/left_join.ddp)); `() inject None` is refused, because `Option`'s `Some` lane has no shape until a value supplies one.
- **There is no NULL.** A missing value is a variant. Declare an enum for it; `() inject None` is refused, because `Option`'s `Some` lane has no shape until a value supplies one.
- **Nothing is durable.** A server's programs and data last as long as its process.
- **One server is one trust domain.** Programs share one command queue and one set of workers: a slow program delays the others, and a program that never finishes or crashes a worker stops them all.
- **Inputs belong to their program.** Another program cannot read an input, only an output. Changing a program's schema is a migration; a stable program that only publishes its inputs, read by programs whose inputs are bound to them, keeps views replaceable.
Expand Down
28 changes: 10 additions & 18 deletions interactive/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# DDIR

[Language reference](LANGUAGE.md) · [Recipes](examples/recipes/README.md) · [Server command reference](server/README.md#server-command-reference)
[Examples](examples/README.md) · [Language reference](LANGUAGE.md) · [Server command reference](server/README.md#server-command-reference)


An interpreted language (and runtime) for computation over continually changing collections.
Expand Down Expand Up @@ -177,12 +177,12 @@ The intent is that the operation of the system is conducted less by humans and k

The examples will feed and read data with textual, row-oriented versions.
This is better done through binary columnar forms, which exist but aren't easy to present.
The program is [`examples/server/programs/access.ddp`](examples/server/programs/access.ddp), and the whole session is [`examples/server/sessions/access.txt`](examples/server/sessions/access.txt).
From `interactive/`, `cargo run --release -p ddir-server < examples/server/sessions/access.txt` runs it end to end.
The program is [`examples/access.ddp`](examples/access.ddp), and the whole session is [`examples/access.txt`](examples/access.txt).
From `interactive/`, `cargo run --release -p ddir-server < examples/access.txt` runs it end to end.

```
# Install access.ddp under the name `demo`.
load demo from examples/server/programs/access.ddp
load demo from examples/access.ddp
```
This binds the name `demo` to the root scope of `access.ddp`.
There are now three inputs we can reference, and the output `access`.
Expand Down Expand Up @@ -329,26 +329,18 @@ The stack aims to stay generic over two things: the expression language and the
Another expression language or backend should plug in at those two places, without the rest of
the stack depending on it.

The `examples/programs/` directory contains example programs, intentionally simple at the moment.
The one executable is the server (the `ddir-server` crate in `server/`, documented in
`server/README.md`; the session scripts in `examples/server/` show it at work); a program runs by
loading it, feeding its inputs at a time, and sealing them past that time. For example, feed a two-edge path and its root on four workers.
`server/README.md`); a program runs by loading it, feeding its inputs at a time, and sealing
them past that time. The session in [`examples/`](examples/README.md) shows it at work. For
example, run the access session on four workers:
```
cd interactive
printf 'load reach from examples/programs/reach.ddp
feed reach 0 0,1 time=0
feed reach 0 1,2 time=0
feed reach 1 0 time=0
seal reach 0 1
seal reach 1 1
peek reach::result as-of 0
exit
' | DDIR_WORKERS=4 cargo run --release -p ddir-server
DDIR_WORKERS=4 cargo run --release -p ddir-server < examples/access.txt
```

For the full command vocabulary — `load`, `feed`, `seal`, `peek`, `tail`, and the rest — see
[`examples/server/README.md`](examples/server/README.md), and `server/README.md` for the
protocol and configuration.
[`server/README.md`](server/README.md#server-command-reference), which also has the protocol
and configuration.

## Differential testing

Expand Down
13 changes: 13 additions & 0 deletions interactive/examples/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Examples

[`access.txt`](access.txt) is a whole session: it loads [`access.ddp`](access.ddp), feeds it, and reads the results.
Run it from `interactive/`, and see [the README](../README.md#operation) for a walk through it:

```sh
cargo run --release -p ddir-server < examples/access.txt
```

The other programs each show one idiom, explained in their comments:
[`top_k`](top_k.ddp), [`retention`](retention.ddp), [`upsert`](upsert.ddp), [`components`](components.ddp), [`shortest_paths`](shortest_paths.ddp), [`triangles`](triangles.ddp).

[`aoc2023`](aoc2023/README.md) is Advent of Code 2023 in DDIR, with its own runner.
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# The access-control example from interactive/README.md. Run from the interactive/ dir:
# cargo run --release -p ddir-server < examples/server/sessions/access.txt
# The access-control session that interactive/README.md walks through.
# Run from interactive/: cargo run --release -p ddir-server < examples/access.txt

# Install access.ddp under the name `demo`.
load demo from examples/server/programs/access.ddp
load demo from examples/access.ddp

# Variants are structural, and written by position:
# inject(0,x) is User x, inject(1,x) is Group x,
Expand Down
19 changes: 19 additions & 0 deletions interactive/examples/components.ddp
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
-- Connected components by label propagation, with prioritized iteration.
--
-- Each node takes the least label among its neighbors. Small labels win, so `delay` holds node
-- n's own label back to round 256 * bit_length(n), letting small labels spread before large ones
-- start work that would be undone. On a 1M-edge graph this took 1.39 s to 0.55 s; some later
-- edits got slower (a bridge in a 300-node path: 37 ms to 3.4 s).

let raw = input "edges" : (int, int); -- an undirected edge, either way round
let edges = raw + (raw | map((a, b) -> (b, a))) | distinct;
let nodes = edges | map((a, _) -> (a, ())) | distinct;

cc: {
-- The inner time is (epoch, round): `0` leaves the epoch alone and sets the round.
let seeds = nodes | map((n, _) -> (n, n)) | delay((_, n) -> (0, (256, n call bit_length) mul));
let proposals = labels | join(edges, (_, (label, dst)) -> (dst, label));
set labels = seeds + proposals | min;
}

output "labels" = cc::labels;
30 changes: 0 additions & 30 deletions interactive/examples/recipes/README.md

This file was deleted.

22 changes: 0 additions & 22 deletions interactive/examples/recipes/connected_components.ddp

This file was deleted.

27 changes: 0 additions & 27 deletions interactive/examples/recipes/connected_components.out

This file was deleted.

21 changes: 0 additions & 21 deletions interactive/examples/recipes/connected_components.txt

This file was deleted.

17 changes: 0 additions & 17 deletions interactive/examples/recipes/left_join.ddp

This file was deleted.

16 changes: 0 additions & 16 deletions interactive/examples/recipes/left_join.out

This file was deleted.

19 changes: 0 additions & 19 deletions interactive/examples/recipes/left_join.txt

This file was deleted.

13 changes: 0 additions & 13 deletions interactive/examples/recipes/retention.ddp

This file was deleted.

Loading
Loading