Skip to content

Late-materialize a grouped node's properties from Lance node tables #99

Closed

Description

@prrao87

Problem

Grouping by a node (WITH forum, collect(friend) AS friends) after a hop that fans out carries every column of the node through the grouping, once per input row rather than once per node. In IC5, about 800K hasMember rows carry the forum's title into 77K groups.

#101 fixed this for in-memory node tables with late materialization. When the label has a node index (unique integer ids), the query groups by the node's id alone, and a NodeLookup operator adds the other columns after the aggregate, once per group. In memory mode that made IC3 52% faster, IC5 7% faster and the complex suite's geomean 0.930.

Lance node tables (lazy mode) still group by every column. By the time rows reach the grouping, an earlier step has already read the node's properties from the dataset, so grouping by them reads nothing more. Turning the lookup on as it stands reads them from disk a second time, and IC1 got 11% slower in lazy mode when #101 tried it.

Idea

Late materialization for Lance node tables. When the node is grouped by, the step before the grouping reads only the id, and the other columns are read from Lance after the aggregate, once per group. For a few groups that's take_rows through the node index. When the groups cover a large share of the table, read the columns once in a sequential scan instead, the way the CSR's target lookup does past 1/8 of the table (#94).

The gain depends on how far the hop fans out compared with how many nodes are left after grouping. A query that groups nearly one row per node gains nothing and pays for a second pass, so planning needs a way to tell the two apart, or a check that shows the cost stays small.

Acceptance

  • Lazy-mode IC5 and IC1 plans and latency, before and after, with results unchanged.
  • Both suites in both modes against main, run with the CONTRIBUTING.md harness and reporting every query. Memory mode should be unchanged.

Activity

  1. changed the title [-]Fetch a node's properties after grouping by it, not before[/-] [+]Late-materialize a grouped node's properties from Lance node tables[/+] on Nov 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions