Skip to content

Store node indexes so engines skip the scan at start-up #117

Open

Description

@prrao87

Problem

With persist_csr=True (#109), an engine loads each relationship type's CSR from a stored file instead of building it. Node indexes still get built at every start. A node index (NodeRows, reported by node_index_stats()) maps each node key of a label to its Lance row id, so a hop can fetch target rows with take_rows. It's built from a scan of every node table, even when the CSR itself is loaded.

On GDC SF1 read lazily, a start that loads every stored CSR takes 0.46 s, against 1.71 s for one that builds. Nobody has measured how much of that 0.46 s goes to node indexes, so measure that first. If it's small, this issue may not be worth doing.

Idea

Store each label's node index the same way #109 stores the CSR: an immutable file per dataset version in the node table's own _lance_csr/ folder, holding keys and row ids. It would be validated against the dataset version and fragments, and brought up to date with a delta from added fragments only. Writes to a node table already invalidate only that label's node index, never the CSR.

Open question: when the node table already has a Lance B-tree scalar index on its ID column, can we use that instead of our own file?

Smaller follow-ups

  • If loading many relationship types still adds up, an engine could load each type's CSR file on the first query that uses it, not at construction. That moves the cost rather than removing it.
  • A future Lance release could garbage-collect files it doesn't recognise, or claim the _lance_csr folder name. Validation would turn either into a rebuild rather than a wrong answer, but it may be worth asking the Lance maintainers to reserve the name.

Split out of #72. Part of #80.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions