[prrao87]

Why

LDBC IC10 was the slowest query where lance-csr trailed other engines on SF1: about 98 ms in memory, against 14 ms for Ladybug. Its graph hops take microseconds. The time went to two places where the planner lowers nested Cypher by flattening and regrouping:

• List comprehensions over collected lists. [p IN posts WHERE ...] unnested each row's list and regrouped on every input column, including the list itself. Each of IC10's 30.8K element rows carried a copy of its friend's whole post list (95 MB through UnnestExec), was hash-joined against the 1M-row Post table, and was hashed back together with that list as a group key.
• Pattern predicates. WHERE NOT (friend)-[:knows]-(person) and (p)-[:postHasTag]->()<-[:hasInterest]-(person) were each answered by planning everything before them a second time as a seed, matching the pattern from that copy, and left-joining a mark back. Each predicate doubles the plan below it, and DataFusion plans and runs every copy: IC10's plan had 203 operators, including 24 CSR expansions for a pattern of about six hops, and planning alone took about 37 ms.

Query	memory	lazy
IC10	97.5 → 24.2 ms (−75%)	130.1 → 34.0 ms (−74%)
IC7	14.4 → 9.8 ms (−32%)	23.0 → 14.6 ms (−36%)
IC14	47.6 → 51.1 ms (+7%; +3% in a direct A/B, see below)	50.5 → 51.0 ms (+1%)
Complex suite, geometric mean	−10.0%	−11.2%

Means of four runs of each build, main (d3c0192) vs this branch; full tables are under Validation. IC7's isNew is a pattern predicate inside CASE. These are the only three queries in either suite whose plans change (IC3's differs only in the order of two conjuncts).

Part of #18: IC10's plan drops from 209 to 63 physical operators. #18 stays open for IC1 and for OPTIONAL MATCH, which still plans its input a second time.

What changed

• A comprehension keeps each input row once. The list is unnested beside its element positions, so an element row holds its item instead of a copy of the list. A list read from a column (such as posts from collect(post)) is no longer a grouping key on the way back: it is rebuilt once per row from its elements, in order, null for a null list and empty for an empty one, and projection pushdown drops it when the comprehension doesn't read it. A list of nodes binds each element through NodeLookup on in-memory node tables instead of a hash join against the whole table.
• New PathExists operator for pattern predicates. A predicate that is one path from a bound node, through anonymous nodes and anonymous relationships of distinct single types, ending at an anonymous or bound node, with no inner WHERE, is answered per input row by walking the CSR with early exit. Every node it reaches must be in its label's node index, matching the relational plan's joins, so edges to missing nodes match nothing. It appends the same true/null mark as before, so filters and CASE above it don't change. Other predicates, labels without a node index and engines without a CSR keep the relational plan, which the logical operator carries as its fallback; relationship-instance numbering stays in step whichever is planned. The column-pruning rule knows the operator reads only its start and end columns.
• Docs: PathExists and the comprehension lowering in the DataFusion internals pages, with explain() plans from a local build, and pattern predicates in the CSR guide's coverage table.

No API change. Results are unchanged; queries of these shapes run faster, and explain() shows PathExistsExec where it applies.

Validation

• New differential test: the same queries with the CSR (PathExists) and without it (relational marking) return identical rows over self-loops, duplicate and mutual edges, posts with no tags, friends with no posts and dangling edges at every hop, and the test asserts which queries walk the CSR. Breaking the end-node comparison or the node-existence check makes it fail.
• New test that a rebuilt list round-trips exactly: null and empty lists, null elements, and a comprehension that reads its own list. Two plan assertions (Rust and Python) now check the physical plan, where the unused rebuild is pruned, instead of the unoptimized logical plan.
• cargo test --workspace --locked, cargo clippy --workspace --all-targets -- -D warnings, cargo fmt --all --check, make test (1665 passed), make lint, docs/check_snippets.py, zensical build --strict and uvx typos@1.26.0 . all pass locally. CI is pending.
• Benchmarks: the CONTRIBUTING.md harness, release wheels of main (d3c0192) and this branch, both suites, both modes, builds alternated, run in two full passes (four runs of each build per mode). All 704 answer checks passed. graph-benchmark-ldbc 6f1b990, graph-benchmark-ldbc-complex 7f45ae2. Apple M5, 24 GiB, macOS 26.6.2.

Reading the deltas: I diffed every query's physical plan between the two wheels. All 30 custom-suite plans and 11 of 14 complex-suite plans are identical, so their deltas are noise, not this change:

• IC12 (+17% in memory) has an identical plan and identical planning time (2.2 ms). It lands in a fast or a slow mode per process: three separate processes each, medians of 60 executions, gave 16.9 / 19.1 / 20.0 ms for a second build of unchanged main, 19.0 / 18.9 / 19.1 for main, and 20.4 / 18.0 / 20.7 for this branch. Four runs per build weren't enough to average that out.
• Custom suite: memory drifted +3.5% and lazy −2.7% over the same unchanged plans, with single main runs differing by up to 43%.
• IC14 is a real, small regression. Its [n IN nodes(path) | n.ID] now goes through the new lowering. In direct A/B runs (three processes per build, medians of 60), it took 49.0–50.6 ms against 47.1–48.4 ms for main and the control build, about +3%, split evenly between planning (24.4 vs 23.6 ms) and execution. The comprehension adds a few expressions to a plan DataFusion already spends 24 ms optimizing; the comprehension itself covers only a handful of paths.

graph-benchmark-ldbc-complex, every query

graph-benchmark-ldbc-complex, memory (4 runs of each build)

Query	Plan changed	main (ms)	branch (ms)	Change
IC1	no	23.25	23.27	+0.1%
IC2	no	2.45	2.49	+1.8%
IC3	filter order only	25.00	25.34	+1.3%
IC4	no	5.85	5.79	-0.9%
IC5	no	77.91	77.93	+0.0%
IC6	no	21.83	22.31	+2.2%
IC7	yes	14.37	9.75	-32.1%
IC8	no	2.28	2.23	-2.1%
IC9	no	37.34	38.28	+2.5%
IC10	yes	97.54	24.24	-75.1%
IC11	no	2.98	3.03	+2.0%
IC12	no	17.70	20.78	+17.4%
IC13	no	1.09	1.10	+0.9%
IC14	yes	47.59	51.09	+7.4%

Geometric mean of branch/main: -10.0%. Spread across the four main runs: median 2.1%, max 5.3%.

graph-benchmark-ldbc-complex, lazy (4 runs of each build)

Query	Plan changed	main (ms)	branch (ms)	Change
IC1	no	31.51	31.38	-0.4%
IC2	no	25.85	25.52	-1.3%
IC3	filter order only	168.95	169.26	+0.2%
IC4	no	13.15	13.24	+0.7%
IC5	no	110.42	110.18	-0.2%
IC6	no	23.54	24.26	+3.1%
IC7	yes	23.00	14.64	-36.4%
IC8	no	3.31	3.33	+0.7%
IC9	no	310.49	319.19	+2.8%
IC10	yes	130.10	34.01	-73.9%
IC11	no	5.44	5.40	-0.7%
IC12	no	20.84	22.38	+7.4%
IC13	no	1.45	1.44	-0.3%
IC14	yes	50.47	50.99	+1.0%

Geometric mean of branch/main: -11.2%. Spread across the four main runs: median 2.6%, max 8.3%.

graph-benchmark-ldbc, every query (no plan changes)

graph-benchmark-ldbc, memory (4 runs of each build)

Query	Plan changed	main (ms)	branch (ms)	Change
q1	no	0.63	0.63	-0.2%
q2	no	0.52	0.53	+0.8%
q3	no	0.64	0.64	-0.4%
q4	no	0.50	0.50	+0.5%
q5	no	0.77	0.78	+1.8%
q6	no	0.42	0.44	+3.0%
q7	no	1.01	1.04	+3.2%
q8	no	0.62	0.63	+1.0%
q9	no	0.80	0.85	+7.0%
q10	no	0.71	0.76	+6.5%
q11	no	1.52	1.60	+5.2%
q12	no	0.74	0.76	+1.7%
q13	no	0.78	0.80	+2.6%
q14	no	0.52	0.53	+2.3%
q15	no	0.55	0.56	+1.5%
q16	no	0.65	0.66	+2.4%
q17	no	0.90	0.92	+1.9%
q18	no	0.69	0.71	+3.5%
q19	no	0.92	0.98	+7.3%
q20	no	2.00	2.18	+9.5%
q21	no	0.76	0.77	+1.1%
q22	no	0.70	0.72	+3.0%
q23	no	0.73	0.74	+1.5%
q24	no	0.68	0.69	+1.2%
q25	no	0.78	0.80	+2.5%
q26	no	0.73	0.81	+10.2%
q27	no	0.85	0.87	+3.0%
q28	no	1.38	1.43	+3.7%
q29	no	0.68	0.73	+7.4%
q30	no	40.84	45.86	+12.3%

Geometric mean of branch/main: +3.5%. Spread across the four main runs: median 4.2%, max 10.3%.

graph-benchmark-ldbc, lazy (4 runs of each build)

Query	Plan changed	main (ms)	branch (ms)	Change
q1	no	1.23	1.17	-4.5%
q2	no	1.51	1.39	-7.8%
q3	no	0.87	0.80	-8.0%
q4	no	0.87	0.78	-10.3%
q5	no	2.36	2.16	-8.4%
q6	no	0.77	0.72	-6.4%
q7	no	1.11	1.08	-2.4%
q8	no	1.79	1.74	-3.3%
q9	no	1.42	1.33	-6.1%
q10	no	0.98	0.92	-5.7%
q11	no	2.09	1.99	-4.5%
q12	no	6.71	6.74	+0.6%
q13	no	1.04	1.05	+0.3%
q14	no	0.70	0.68	-2.3%
q15	no	0.77	0.75	-2.9%
q16	no	1.12	1.08	-3.5%
q17	no	1.42	1.39	-2.2%
q18	no	1.15	1.13	-1.7%
q19	no	1.38	1.35	-2.2%
q20	no	16.79	16.67	-0.7%
q21	no	0.85	0.83	-1.9%
q22	no	0.92	0.92	-0.1%
q23	no	1.24	1.22	-0.9%
q24	no	1.05	1.05	-0.2%
q25	no	1.24	1.19	-4.3%
q26	no	1.07	1.08	+0.6%
q27	no	1.29	1.31	+0.9%
q28	no	2.04	2.09	+2.0%
q29	no	1.02	1.03	+0.9%
q30	no	41.56	43.30	+4.2%

Geometric mean of branch/main: -2.7%. Spread across the four main runs: median 9.0%, max 42.9%.

🤖 Generated with Claude Code, reviewed by @prrao87.

[cloudflare-workers-and-pages]

Deploying lance-csr with    Cloudflare Pages

Latest commit:	4e3c88c
Status:	✅  Deploy successful!
Preview URL:	https://1515eb95.lance-csr.pages.dev
Branch Preview URL:	https://perf-in-place-pattern-predic.lance-csr.pages.dev

View logs