Repository navigation
Shrink plans for comprehension-heavy queries such as IC10 #18
Description
Problem
Queries built from pattern comprehensions, OPTIONAL MATCH, collect() and chained WITH produce very large physical plans. Two of them, IC10 and IC1, are where lance-csr trails other engines most; IC14 has the same plan growth, although it is already faster than the other engines.
Evidence
Physical operator counts on SF1, from explain():
| Query | Operators | Notable | lance-csr in memory | Fastest other engine |
|---|---|---|---|---|
| IC10 | 217 | 38 AggregateExec, 32 CsrExpandExec |
105.0 ms | 13.7 ms (Ladybug) |
| IC14 | 225 | 49 RepartitionExec |
64.4 ms | 522.6 ms (Neo4j) |
| IC1 | 164 | 20 AggregateExec |
26.8 ms | 13.5 ms (Kuzu) |
IC10 collects each friend's posts and counts those that share a tag with the person, with size([p IN posts WHERE (p)-[:postHasTag]->()<-[:hasInterest]-(person)]). An earlier change (seeding comprehensions from the original rows) cut IC14 from 641 to 225 operators and from 188 to 62 ms, so there is precedent for large wins here.
Suggested fix
Start with IC10: read its logical plan and find where the comprehension and the OPTIONAL MATCH ... collect() are planned as separate joins and aggregations over the same rows. Candidates are evaluating a pattern predicate inside a list comprehension as a semi-join over the unnested list, and merging aggregations that group by the same keys.
Acceptance criteria
- IC10's operator count and latency before and after, in the pull request, with results unchanged.
- No regression on the other IC queries or the 30-query workload.
- Any new planning rule has focused tests.
Identified by profiling the LDBC IC benchmark; IC10's plan has not been analysed in detail.
Activity
perf(cypher): count unmapped comprehensions without list aggregation (#…
feat: integrate cypher, csr and engine work for 0.3.0 (#55)
prrao87 commentedon Oct 29, 2026
Partial progress from #38, which counts
size([x IN list WHERE ...])without building the list. On currentmain(Apple M5, SF1, in memory), IC10 is down from 217 to 201 operators (AggregateExec38 → 32,CsrExpandExec32 → 24) and from 105.0 to 97.8 ms, still about 7x slower than Ladybug's 13.7 ms. IC1 is unchanged at 164 operators. The suggested fixes, a semi-join for the pattern predicate and merging aggregations with the same keys, are still open.prrao87 commentedon Nov 2, 2026
Progress from #108, which implements the semi-join suggested here and also stops list comprehensions from copying each row's list onto every element. Measured on SF1, Apple M5, against
mainat d3c0192, with four harness runs of each build per mode:mainAggregateExec/CsrExpandExec/HashJoinExecmemorylazyHow it gets there:
(p)-[:postHasTag]->()<-[:hasInterest]-(person)andNOT (friend)-[:knows]-(person)are answered per row by a newPathExistsoperator that walks the CSR, instead of re-planning everything before them as a seed and joining a mark back. Each predicate used to double the plan beneath it.[p IN posts WHERE ...]unnests the list beside its positions, rebuilds a list read from a column once per row instead of grouping on it, and binds node elements throughNodeLookupinstead of a hash join against the whole node table.Still open here:
OPTIONAL MATCHstill plans its input a second time as the seed of its optional side and joins back. It is now the largest remaining duplication in IC10. An outer mode for single-hop CSR expansion would remove it the same wayPathExistsremoved the predicate copies.array_agg, which has no grouped fast path. Dropping the order saved about 9 ms on IC10 in an experiment, but is incorrect in general.memorymode in direct A/B runs (0.8 ms planning, 0.8 ms execution). Its[n IN nodes(path) | n.ID]now takes the new lowering, which adds a few expressions to an already large plan.main, identical with perf(planner): walk pattern predicates on the csr and keep comprehension rows once #108). Neither change applies to it, and merging aggregations with the same keys is still open.🤖 Posted by Claude Code
perf(csr): expand one-hop OPTIONAL MATCH as an outer CSR hop (#118)