perf(query): official IC14 text with reduce() is over 1000x slower than the benchmark port #123

Summary

The official LDBC SNB Interactive IC14 query ("Trusted connection paths"), as written for Neo4j with reduce() and pattern comprehensions, runs correctly on lance-csr but takes about 2 minutes on SF1. The ported version the benchmark uses takes under 0.1 s. Both return the same 58 rows.

All runs used memory mode and use_csr=True, on one Apple Silicon Mac. The rows match once ties in pathWeight are ignored; the query orders only by pathWeight.

Context

graph-benchmark-ldbc-complex runs IC1 to IC13 with Neo4j's Cypher text unchanged, but rewrites IC14. Its docs, and docs/benchmarks.md here, say the rewrite is needed because lance-csr doesn't implement reduce(). That's out of date: reduce() has been supported since 0.3.0 (#55). Now the reason to keep the port is speed.

The official text:

MATCH path = allShortestPaths((person1:Person {ID: $person1Id})-[:knows*0..]-(person2:Person {ID: $person2Id}))
WITH collect(path) AS paths
UNWIND paths AS path
WITH path, relationships(path) AS rels_in_path
WITH
    [n IN nodes(path) | n.ID] AS personIdsInPath,
    [r IN rels_in_path |
        reduce(w = 0.0, v IN [
            (a:Person)<-[:commentHasCreator]-(:Comment)-[:replyOfPost]->(:Post)-[:postHasCreator]->(b:Person)
            WHERE (a.ID = startNode(r).ID AND b.ID = endNode(r).ID) OR (a.ID = endNode(r).ID AND b.ID = startNode(r).ID)
            | 1.0] | w + v)
    ] AS weight1,
    [r IN rels_in_path |
        reduce(w = 0.0, v IN [
            (a:Person)<-[:commentHasCreator]-(:Comment)-[:replyOfComment]->(:Comment)-[:commentHasCreator]->(b:Person)
            WHERE (a.ID = startNode(r).ID AND b.ID = endNode(r).ID) OR (a.ID = endNode(r).ID AND b.ID = startNode(r).ID)
            | 0.5] | w + v)
    ] AS weight2
WITH
    personIdsInPath,
    reduce(w = 0.0, v IN weight1 | w + v) AS w1,
    reduce(w = 0.0, v IN weight2 | w + v) AS w2
RETURN personIdsInPath, (w1 + w2) AS pathWeight
ORDER BY pathWeight DESC

Parameters: {"person1Id": 1131, "person2Id": 19791209300775}.

Reproduce

1. In graph-benchmark-ldbc-complex, build the SF1 Lance graph with lance_csr/build_graph.py.
2. Build the engine with engine() from lance_csr/query.py. Run the text above through its _execute helper, then run PORTS[14] with the same parameters, and time both.

Done when

The official IC14 text runs in about the same time as the port, so the benchmark can run IC14 unchanged like the other 13 queries. Then update the benchmark's README and query.py docstring, and docs/benchmarks.md here.