feat: NeuG embedded graph DB, Cypher CLI, and GDS Leiden clustering (opt-in) - #2895
feat: NeuG embedded graph DB, Cypher CLI, and GDS Leiden clustering (opt-in)#2895BingqingLyu wants to merge 1 commit into
Conversation
…opt-in) Add an optional NeuG-backed pipeline alongside the existing NetworkX one: - storage.py: graph.db layer — single-table schema, COPY FROM bulk and incremental ingest with source pruning, backup-on-write protection - cypher command (CLI + MCP): ad-hoc Cypher queries against graph.db - GDS Leiden clustering: --resolution, --cluster-on-files, hub-based community labels, cohesion, god nodes, surprising connections - delta-cluster: incremental community analysis via freeze-assign Leiden; --baseline seeds from an external clustering (old communities frozen, new nodes assigned on top) - opt-in by design: enabled via GRAPHIFY_NEUG=1 or an existing graph.db; the default extract path is unchanged - tests: test_storage.py, test_cypher_cli.py; neug extra in pyproject and uv.lock
![graphify-labs[bot]](https://avatars.githubusercontent.com/in/4356270?s=60&v=4){kind=small}
There was a problem hiding this comment.
Graphify reviewed this change.
Worth a look — the grounded gate found no coupling regressions or blocking issues, but 5 advisory finding(s) below merit a look before merge.
Formal verification. 1 change(s) tested, no difference found (not proven).
Graphify review — findings
Adds an opt-in NeuG embedded-graph-DB pipeline: introduces graphify cypher and graphify delta-cluster CLI commands, a GRAPHIFY_NEUG=1 path in extract that builds graph.db and runs GDS Leiden clustering, plus a --cluster-on-files flag. Wires up storage.py (init/ingest/export, god-node/god-file/surprising-connection queries, Leiden subgraph clustering, freeze-assign delta analysis) and extends serve.py with NeuG-backed context filtering and graph-stats/query tools. Updates README and ARCHITECTURE docs for the new neug extra and commands, and adds storage tests.
Worth a look
- --allow-partial guard removed from no-cluster overwrite path —
graphify/cli.py· Escalate · high- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review Execution auto-disposal is off for this run; enable it (with sandbox isolation) to have Graphify try to confirm or refute this automatically.
- NeuG clustered global merge path no longer calls global_add —
graphify/cli.py:4004· Escalate · high- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review Execution auto-disposal is off for this run; enable it (with sandbox isolation) to have Graphify try to confirm or refute this automatically.
- --global no longer merges in the Neug clustered extract path —
graphify/cli.py:4237· Escalate · high- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review Execution auto-disposal is off for this run; enable it (with sandbox isolation) to have Graphify try to confirm or refute this automatically.
- MCP tool executes arbitrary client-supplied Cypher against graph.db —
graphify/serve.py:1983· Escalate · high- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review Execution auto-disposal is off for this run; enable it (with sandbox isolation) to have Graphify try to confirm or refute this automatically.
- --clear-ast no longer propagated when saving no-cluster manifest —
graphify/cli.py· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review Execution auto-disposal is off for this run; enable it (with sandbox isolation) to have Graphify try to confirm or refute this automatically.
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 706 functions depend on the 331 functions this change touches.
Health — this change adds coupling hotspots:
- new:
dispatch_command()— 2 callers, 124 callees - new:
_query_graph_text()— 20 callers, 9 callees - new:
_score_query()— 15 callers, 5 callees - new:
_query_terms()— 20 callers, 3 callees - new:
delta_analyze()— 5 callers, 10 callees - new:
run_benchmark()— 16 callers, 3 callees - new:
_stale_graph_sources()— 7 callers, 6 callees - new:
_build_server()— 2 callers, 18 callees - …and 25 more — each is listed as a finding
Verification — 706 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 706 function(s) in the blast radius were not formally verified this run
Formal verification
Could not verify: Could not verify dispatch\_command.
The verifier did not have enough to check dispatch\_command, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: not verifiable: all 23 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly SystemExit — names the real obstacle, not a sampling gap)
No difference found (not proven): No behavior difference found in \_run\_cli (not a proof).
The verifier ran both versions of \_run\_cli on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.
Guarantee: Empirical: differential testing (both versions run on many generated inputs). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.
Note: An input the sampler did not try could still differ.
Could not verify: Could not verify \_build\_server.
The verifier did not have enough to check \_build\_server, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: not verifiable: all 23 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly ImportError — names the real obstacle, not a sampling gap)
· 14 grounded finding(s) anchored inline below; 19 more finding(s) on lines outside this diff (see the check run).
| return node_types | ||
|
|
||
|
|
||
| def ingest_extraction( |
There was a problem hiding this comment.
ingest_extraction()
12 callers depend on it (afferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| shutil.copy2(node_csv, dest_dir / f"{tag}_nodes.csv") | ||
|
|
||
|
|
||
| def cluster_on_files( |
There was a problem hiding this comment.
cluster_on_files()
high coupling complexity (Ca·Ce = 25).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| # --------------------------------------------------------------------------- | ||
|
|
||
|
|
||
| def find_god_nodes(conn: object, top_n: int = 10) -> list[dict]: |
There was a problem hiding this comment.
find_god_nodes()
high coupling complexity (Ca·Ce = 12).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| return score, reasons | ||
|
|
||
|
|
||
| def find_surprising_connections( |
There was a problem hiding this comment.
find_surprising_connections()
high coupling complexity (Ca·Ce = 12).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| # --------------------------------------------------------------------------- | ||
|
|
||
|
|
||
| def cluster_by_neug( |
There was a problem hiding this comment.
cluster_by_neug()
fans out to 9 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| _close(db, conn) | ||
|
|
||
|
|
||
| def test_find_surprising_connections(tmp_db): |
There was a problem hiding this comment.
test_find_surprising_connections()
fans out to 6 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| _close(db, conn) | ||
|
|
||
|
|
||
| def test_label_communities_by_hub(tmp_db): |
There was a problem hiding this comment.
test_label_communities_by_hub()
fans out to 6 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| # --- incremental delta analysis (freeze-assign leiden) --- | ||
|
|
||
|
|
||
| def test_run_leiden_freeze_assign(tmp_db): |
There was a problem hiding this comment.
test_run_leiden_freeze_assign()
fans out to 6 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| _close(db, conn) | ||
|
|
||
|
|
||
| def test_run_leiden_freeze_assign_resolution(tmp_db): |
There was a problem hiding this comment.
test_run_leiden_freeze_assign_resolution()
fans out to 6 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| assert changes["dissolved_communities"][0]["old_size"] == 3 | ||
|
|
||
|
|
||
| def test_delta_analyze(tmp_db): |
There was a problem hiding this comment.
test_delta_analyze()
fans out to 6 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
1 participant
Summary
graphify cypher) and MCP server (cypher_querytool)graph.dbdelta-cluster: incremental community analysis command;--baselineseeds from an external clusteringGRAPHIFY_NEUG=1or an existinggraph.db; the default extract path is unchangedMotivation
Graphify currently uses NetworkX +
graph.jsonas its core graph storage. This architecture has bottlenecks:graph.json→ merge → rebuild → re-serialize (O(full graph) even for single-file changes)Why NeuG?
NeuG is a lightweight embedded graph database (C++ core, Python bindings):
pip install neugis all it takes)Architecture
Dual-engine coexistence, each independently consuming extraction data:
When
GRAPHIFY_NEUG=1is set (orgraph.dbalready exists), the NeuG pipeline also runs:graph.dbvia COPY FROM (bulk) or MERGE (incremental)graph.dbgraph.jsonfromgraph.db(not dual-write) — ensures downstream tools (wiki generation, HTML visualization, community labeling) continue to work unchanged, maintaining full backward compatibilityChanges
graphify/storage.pygraphify/cli.pydelta-clustersubcommand, segfault guard for tiny graphsgraphify/__main__.pygraphify cypherCLI commandgraphify/serve.pycypher_queryMCP tool for AI agentsgraphify/llm.pypyproject.tomlneug>=0.1.3optional dependency (neugextra +allextra)README.mdARCHITECTURE.mdstorage.pymodule descriptiontests/test_storage.py,test_cypher_cli.py)Usage
Test Plan
pytest tests/test_storage.py tests/test_cypher_cli.py -v— all tests passedtree-sitter-hcl)cypher_querytool end-to-end verifiedgraphify extract .runs normally (silent skip)Note
This PR builds on the NeuG integration proposed in #1056. While #1056 established the core storage layer (graph.db, Cypher queries, MCP tool), this PR adds GDS extension support for native and incremental community detection.