Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions interactive/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -371,11 +371,12 @@ I would not recommend relying on any of this yet.

## Plan without running

`cargo run -p interactive --bin ddir-plan -- [name=]file.ddp [[name=]more.ddp ...]` prints
`cargo run -p interactive --bin ddir-plan -- [--bind label=trace ...] [name=]file.ddp [[--bind label=trace ...] [name=]more.ddp ...]` prints
lowering, optimization, projection, and as-of-drop stages. `--passes LIST` switches
passes off as the server's `DDIR_PASSES` does; a switched-off stage is shown unchanged. Earlier files supply
their inferred output shapes to later files, as `<name>::<output>`, where `name` is given or is the file's stem. No dataflow or generated source is
installed. A missing provider is unresolved even if its import declares a shape;
their inferred output shapes to later files, as `<name>::<output>`, where `name` is given or is the file's stem.
`--bind label=trace` binds an input of the next file to such a trace, as a load does. No dataflow or generated source is
installed. A missing provider is unresolved even though the bound input declares a shape;
the tool prints the partial report and exits unsuccessfully.

`ddir-plan --timings file.ddp` adds wall times, call counts, rewrite counts, and
Expand Down
4 changes: 2 additions & 2 deletions interactive/bench/ldbc/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,11 +93,11 @@ inclusive interval. Reference implementations:
[BI5](https://github.com/ldbc/ldbc_snb_bi/blob/47dd38b40844ecdb0e42e5a610c369535304786d/umbra/queries/bi-5.sql),
[BI11](https://github.com/ldbc/ldbc_snb_bi/blob/47dd38b40844ecdb0e42e5a610c369535304786d/umbra/queries/bi-11.sql).

All selected plans are installed once and import a shared graph program
All selected plans are installed once and read a shared graph program
(`snb/indexes.py`). It publishes each relation by primary key, secondary indexes
from foreign keys, friendships in both directions and each message's thread root;
its load and maintenance costs are included. Even isolated
controls retain this fixed substrate. This uses existing named-import semantics;
controls retain this fixed substrate. Each plan's named inputs are bound to the graph's outputs at load;
it does not introduce native Corgi trace sharing or a new prepared-query feature.

Each round runs requests on the initial graph, retracts those bindings, removes
Expand Down
4 changes: 2 additions & 2 deletions interactive/bench/ldbc/SNB.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,9 +118,9 @@ bank entries are for transient request batches, not simultaneous BI bindings.
| `maintained` | Standing | Standing | All bound answers maintained and checked |

All query programs are installed once, with the shared graph program, before
any data arrives. Every query imports the arrangements the graph program
any data arrives. Every query reads the arrangements the graph program
publishes (`snb/indexes.py`: each relation by primary key, secondary indexes,
friendships and thread roots) and has a separate request input. Imports read
friendships and thread roots) and has a separate request input. Bound inputs read
the published arrangement in place. Parameter-independent parts of an installed
query remain as dataflows even when its request input is empty; their
maintenance is included. Other derived relations are not factored across
Expand Down
10 changes: 6 additions & 4 deletions interactive/bench/ldbc/authored/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,10 @@ An authored plan is a drop-in replacement for the generated one:
query's `@query(...)` declaration in `snb/queries.py`, prefixed by `rid`. For
example IC3 is `(rid, person, country X, country Y, start, days)`. Byte strings
are `List(int)`.
- **Graph.** Plans import the arrangements the suite's shared graph program publishes
(`snb/indexes.py`), and join against them directly:
- **Graph.** Plans read the arrangements the suite's shared graph program publishes
(`snb/indexes.py`), and join against them directly.
Each is a named input, its label the output's name with `.` written `_` (`message_by_creator`
for `graph::message.by_creator`), and the driver binds it at load (`bench/ldbc/client.py`):
- **primary keys**, `graph::person`, `graph::message`, `graph::forum`, `graph::tag`,
`graph::tagclass`, `graph::place`, `graph::org`: `(id, (the other fields))`, in
`snb/data.py`'s field order;
Expand All @@ -31,7 +33,7 @@ An authored plan is a drop-in replacement for the generated one:
`graph::message.root` (message, root post) and `graph::message.by_root`.

Every query's server shares these, so a plan pays only for what it builds itself.
A `map` or `filter` on an import makes a private copy; the plans do that only for
A `map` or `filter` on a graph input makes a private copy; the plans do that only for
parts kept for all requests, and each header says which.
- **Answer.** `output "answer"`, read as `<query>::answer`, is arranged as `((rid, rank), (fields…))`,
where rank is the 0-based position in the query's order and the fields are the
Expand Down Expand Up @@ -186,4 +188,4 @@ Two conventions apply throughout:

Recurring fragments (where people live, the interaction weights) are repeated in
each file, so that every plan is complete on its own; friendships and threads are
imported.
read from the graph.
2 changes: 1 addition & 1 deletion interactive/bench/ldbc/authored/plans/bi01.ddp
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
-- every message, so this plan scans the message arrangement rather than looking up in it.

let request = input 0 : ((int, int), ()); -- (rid, end time in ms)
let message = import "graph::message" : (int, (int, int, int, int, List(int), List(int), int, List(int), int, int, int, int, int));
let message = input "message" : (int, (int, int, int, int, List(int), List(int), int, List(int), int, int, int, int, int));
-- (id, (kind, created, creator, country, content, image, length, language, forum, parent, year, month, day))

-- Length categories: under 40, under 80, under 160, longer.
Expand Down
8 changes: 4 additions & 4 deletions interactive/bench/ldbc/authored/plans/bi02.ddp
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,11 @@
-- each message's date in the message arrangement.

let request = input 0 : ((int, int, List(int)), ()); -- (rid, date, class name)
let message = import "graph::message" : (int, (int, int, int, int, List(int), List(int), int, List(int), int, int, int, int, int));
let message = input "message" : (int, (int, int, int, int, List(int), List(int), int, List(int), int, int, int, int, int));
-- (id, (kind, created, creator, country, content, image, length, language, forum, parent, year, month, day))
let tag = import "graph::tag" : (int, (List(int), int)); -- (id, (name, class))
let tagged = import "graph::mtag.by_tag" : (int, int); -- (tag, message)
let class_id = import "graph::tagclass.by_name" : (List(int), int); -- (name, id)
let tag = input "tag" : (int, (List(int), int)); -- (id, (name, class))
let tagged = input "mtag_by_tag" : (int, int); -- (tag, message)
let class_id = input "tagclass_by_name" : (List(int), int); -- (name, id)

-- ((class, day), (tag name, created)) for every tagged message, and each class's days.
let byday = tag | join(tagged, (_, ((name, cls), m)) -> (m, (cls, name)))
Expand Down
20 changes: 10 additions & 10 deletions interactive/bench/ldbc/authored/plans/bi03.ddp
Original file line number Diff line number Diff line change
Expand Up @@ -4,17 +4,17 @@
-- messages, and each message's thread root, whose forum is the thread's.

let request = input 0 : ((int, List(int), List(int)), ()); -- (rid, class name, country name)
let message = import "graph::message" : (int, (int, int, int, int, List(int), List(int), int, List(int), int, int, int, int, int));
let message = input "message" : (int, (int, int, int, int, List(int), List(int), int, List(int), int, int, int, int, int));
-- (id, (kind, created, creator, country, content, image, length, language, forum, parent, year, month, day))
let forum = import "graph::forum" : (int, (List(int), int, int)); -- (id, (title, created, moderator))
let moderates = import "graph::forum.by_moderator" : (int, int); -- (person, forum)
let residents = import "graph::person.by_city" : (int, int); -- (city, person)
let place_id = import "graph::place.by_name" : (List(int), int); -- (name, id)
let within = import "graph::place.by_parent" : (int, int); -- (parent, place)
let class_id = import "graph::tagclass.by_name" : (List(int), int); -- (name, id)
let in_class = import "graph::tag.by_class" : (int, int); -- (class, tag)
let tagged = import "graph::mtag.by_tag" : (int, int); -- (tag, message)
let root_of = import "graph::message.root" : (int, int); -- (message, thread root)
let forum = input "forum" : (int, (List(int), int, int)); -- (id, (title, created, moderator))
let moderates = input "forum_by_moderator" : (int, int); -- (person, forum)
let residents = input "person_by_city" : (int, int); -- (city, person)
let place_id = input "place_by_name" : (List(int), int); -- (name, id)
let within = input "place_by_parent" : (int, int); -- (parent, place)
let class_id = input "tagclass_by_name" : (List(int), int); -- (name, id)
let in_class = input "tag_by_class" : (int, int); -- (class, tag)
let tagged = input "mtag_by_tag" : (int, int); -- (tag, message)
let root_of = input "message_root" : (int, int); -- (message, thread root)

-- Forums moderated by someone living in a city of the country.
let local = request | map(((rid, _, country), ()) -> (country, rid))
Expand Down
18 changes: 9 additions & 9 deletions interactive/bench/ldbc/authored/plans/bi04.ddp
Original file line number Diff line number Diff line change
Expand Up @@ -5,17 +5,17 @@
-- creation date and ranks, then looks up the winners' members, posts and threads.

let request = input 0 : ((int, int), ()); -- (rid, date)
let message = import "graph::message" : (int, (int, int, int, int, List(int), List(int), int, List(int), int, int, int, int, int));
let message = input "message" : (int, (int, int, int, int, List(int), List(int), int, List(int), int, int, int, int, int));
-- (id, (kind, created, creator, country, content, image, length, language, forum, parent, year, month, day))
let person = import "graph::person" : (int, (List(int), List(int), List(int), int, int, List(int), List(int), int, int, int, int));
let person = input "person" : (int, (List(int), List(int), List(int), int, int, List(int), List(int), int, int, int, int));
-- (id, (first, last, gender, birthday, created, ip, browser, city, birth month, birth day, creation month))
let forum = import "graph::forum" : (int, (List(int), int, int)); -- (id, (title, created, moderator))
let place = import "graph::place" : (int, (List(int), List(int), int)); -- (id, (name, type, parent))
let residents = import "graph::person.by_city" : (int, int); -- (city, person)
let members = import "graph::member.by_forum" : (int, (int, int)); -- (forum, (person, joined))
let joins = import "graph::member.by_person" : (int, (int, int)); -- (person, (forum, joined))
let posts_in = import "graph::message.by_forum" : (int, int); -- (forum, post)
let thread = import "graph::message.by_root" : (int, int); -- (root, message)
let forum = input "forum" : (int, (List(int), int, int)); -- (id, (title, created, moderator))
let place = input "place" : (int, (List(int), List(int), int)); -- (id, (name, type, parent))
let residents = input "person_by_city" : (int, int); -- (city, person)
let members = input "member_by_forum" : (int, (int, int)); -- (forum, (person, joined))
let joins = input "member_by_person" : (int, (int, int)); -- (person, (forum, joined))
let posts_in = input "message_by_forum" : (int, int); -- (forum, post)
let thread = input "message_by_root" : (int, int); -- (root, message)

-- Each person's country, and each forum's popularity (negated, so min is max).
let country_of = place | join(residents, (_, ((_, _, country), who)) -> (who, country));
Expand Down
10 changes: 5 additions & 5 deletions interactive/bench/ldbc/authored/plans/bi05.ddp
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,12 @@
-- looked up from the request: the tag's messages, their authors, replies and likes.

let request = input 0 : ((int, List(int)), ()); -- (rid, tag name)
let message = import "graph::message" : (int, (int, int, int, int, List(int), List(int), int, List(int), int, int, int, int, int));
let message = input "message" : (int, (int, int, int, int, List(int), List(int), int, List(int), int, int, int, int, int));
-- (id, (kind, created, creator, country, content, image, length, language, forum, parent, year, month, day))
let tag_id = import "graph::tag.by_name" : (List(int), int); -- (name, id)
let tagged = import "graph::mtag.by_tag" : (int, int); -- (tag, message)
let replies = import "graph::message.by_parent" : (int, int); -- (message, reply)
let liked_by = import "graph::likes.by_message" : (int, (int, int)); -- (message, (person, created))
let tag_id = input "tag_by_name" : (List(int), int); -- (name, id)
let tagged = input "mtag_by_tag" : (int, int); -- (tag, message)
let replies = input "message_by_parent" : (int, int); -- (message, reply)
let liked_by = input "likes_by_message" : (int, (int, int)); -- (message, (person, created))

-- The request's tagged messages and their authors: (message, (rid, author)).
-- Tag names are unique and (message, tag) pairs are a set, so each message appears once.
Expand Down
8 changes: 4 additions & 4 deletions interactive/bench/ldbc/authored/plans/bi06.ddp
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,11 @@
-- message's author); the rest is looked up from the request.

let request = input 0 : ((int, List(int)), ()); -- (rid, tag name)
let message = import "graph::message" : (int, (int, int, int, int, List(int), List(int), int, List(int), int, int, int, int, int));
let message = input "message" : (int, (int, int, int, int, List(int), List(int), int, List(int), int, int, int, int, int));
-- (id, (kind, created, creator, country, content, image, length, language, forum, parent, year, month, day))
let tag_id = import "graph::tag.by_name" : (List(int), int); -- (name, id)
let tagged = import "graph::mtag.by_tag" : (int, int); -- (tag, message)
let liked_by = import "graph::likes.by_message" : (int, (int, int)); -- (message, (person, created))
let tag_id = input "tag_by_name" : (List(int), int); -- (name, id)
let tagged = input "mtag_by_tag" : (int, int); -- (tag, message)
let liked_by = input "likes_by_message" : (int, (int, int)); -- (message, (person, created))

-- The request's tagged messages and their authors (one tag; (message, tag) is a set).
let written = request | map(((rid, name), ()) -> (name, rid))
Expand Down
10 changes: 5 additions & 5 deletions interactive/bench/ldbc/authored/plans/bi07.ddp
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,11 @@
-- the tag's messages, their replies, and the replies' tags.

let request = input 0 : ((int, List(int)), ()); -- (rid, tag name)
let tag = import "graph::tag" : (int, (List(int), int)); -- (id, (name, class))
let tag_id = import "graph::tag.by_name" : (List(int), int); -- (name, id)
let tagged = import "graph::mtag.by_tag" : (int, int); -- (tag, message)
let tags_of = import "graph::mtag.by_message" : (int, int); -- (message, tag)
let replies = import "graph::message.by_parent" : (int, int); -- (message, reply)
let tag = input "tag" : (int, (List(int), int)); -- (id, (name, class))
let tag_id = input "tag_by_name" : (List(int), int); -- (name, id)
let tagged = input "mtag_by_tag" : (int, int); -- (tag, message)
let tags_of = input "mtag_by_message" : (int, int); -- (message, tag)
let replies = input "message_by_parent" : (int, int); -- (message, reply)

-- Messages carrying the request's tag (one tag; (message, tag) is a set).
let marked = request | map(((rid, name), ()) -> (name, rid))
Expand Down
10 changes: 5 additions & 5 deletions interactive/bench/ldbc/authored/plans/bi08.ddp
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,12 @@
-- the scored people's friends.

let request = input 0 : ((int, List(int), int, int), ()); -- (rid, tag name, start, end)
let message = import "graph::message" : (int, (int, int, int, int, List(int), List(int), int, List(int), int, int, int, int, int));
let message = input "message" : (int, (int, int, int, int, List(int), List(int), int, List(int), int, int, int, int, int));
-- (id, (kind, created, creator, country, content, image, length, language, forum, parent, year, month, day))
let friend = import "graph::friend" : (int, (int, int)); -- (person, (friend, since))
let tag_id = import "graph::tag.by_name" : (List(int), int); -- (name, id)
let tagged = import "graph::mtag.by_tag" : (int, int); -- (tag, message)
let fans_of = import "graph::interest.by_tag" : (int, int); -- (tag, person)
let friend = input "friend" : (int, (int, int)); -- (person, (friend, since))
let tag_id = input "tag_by_name" : (List(int), int); -- (name, id)
let tagged = input "mtag_by_tag" : (int, int); -- (tag, message)
let fans_of = input "interest_by_tag" : (int, int); -- (tag, person)

let tag_rid = request | map(((rid, name, _, _), ()) -> (name, rid)) | join(tag_id, (_, (rid, t)) -> (t, rid));
let fans = tag_rid | join(fans_of, (_, (rid, who)) -> ((rid, who), 100));
Expand Down
8 changes: 4 additions & 4 deletions interactive/bench/ldbc/authored/plans/bi09.ddp
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,12 @@
-- (no published index is keyed by day).

let request = input 0 : ((int, int, int), ()); -- (rid, start, end)
let message = import "graph::message" : (int, (int, int, int, int, List(int), List(int), int, List(int), int, int, int, int, int));
let message = input "message" : (int, (int, int, int, int, List(int), List(int), int, List(int), int, int, int, int, int));
-- (id, (kind, created, creator, country, content, image, length, language, forum, parent, year, month, day))
let person = import "graph::person" : (int, (List(int), List(int), List(int), int, int, List(int), List(int), int, int, int, int));
let person = input "person" : (int, (List(int), List(int), List(int), int, int, List(int), List(int), int, int, int, int));
-- (id, (first, last, gender, birthday, created, ip, browser, city, birth month, birth day, creation month))
let root_of = import "graph::message.root" : (int, int); -- (message, thread root)
let thread = import "graph::message.by_root" : (int, int); -- (root, message)
let root_of = input "message_root" : (int, int); -- (message, thread root)
let thread = input "message_by_root" : (int, int); -- (root, message)

-- Messages by day, and the request's window days.
let byday = message | map((id, (kind, created, who, _, _, _, _, _, _, _, _, _, day)) -> (day, (id, kind, created, who)));
Expand Down
Loading
Loading