perf(gfql): route engine=auto to native polars for polars-frame graphs

[lmeyerov]

Summary

resolve_engine(AUTO) maps polars frames to Engine.PANDAS — that branch predates Engine.POLARS — so g.gfql(query) on a polars-frame graph silently bridges every query onto the generic pandas path and hands pandas frames back.

This routes engine='auto' (the default) to the native polars engine when both bound frames are polars, with an honest-NotImplementedError fallback to the legacy AUTO path. The fallback is permitted here because the user pinned no engine; the no-silent-fallback contract still applies to an explicit engine=.

This is a user-visible default-behaviour change, not a bench tweak. Scope is deliberately gfql-only (one guard in gfql()): changing resolve_engine itself would ripple into plotting/util paths that only handle pandas/cudf.

The measurements in the original body (a 2k-node microbench, "~3x / ~13x") have been replaced. Everything below is measured on dgx-spark under the exclusive perf lock, position-balanced over 8 slots, against origin/master 233b64c8.

What actually changes, at the surface

	before	after
frame type out of AUTO on a polars graph	pandas	polars (frames in = frames out)
nullable integer columns	upcast to float64 (30 → 30.0)	stay Int64 with nulls
nullable boolean columns	object	Boolean
shapes the native engine declines	ran on the generic path	still do, via the fallback (+0.23 ms)
queries carrying a policy	generic path	unchanged — deliberately not routed (see below)
explicit engine='pandas' / 'polars' / cudf	—	unchanged
graphs whose node and edge frames are not BOTH polars	—	unchanged

Perf: both directions, per-slot, with an engagement receipt

Matched graph-benchmark q1-q9, run through g.gfql(<cypher>) with no engine argument on polars-materialized frames — the surface this PR moves, and the one the committed lanes cannot see because they all pin engine=. Arms: A = 233b64c8, B = this branch. One session, slot order A B B A B A A B, --gpus all, perf lock held. 20k: 7 runs / 3 warmup; 100k: 5 / 2.

frames is the engagement receipt: the module of the returned _nodes frame, recorded per query. pandas → polars means the routing engaged on that cell. A cell that reported pandas on arm B would mean the routing did not engage there and its number would mean nothing.

20k (20,030 nodes / 260,107 edges)

q	A per-slot (ms)	B per-slot (ms)	A med	B med	Δ	verdict	frames
q1	40.17 40.52 41.31 41.80	12.77 14.24 14.49 14.49	41.31	14.49	−64.9%	FASTER	pandas→polars
q2	69.04 70.03 70.30 70.94	17.29 17.51 17.77 19.49	70.30	17.77	−74.7%	FASTER	pandas→polars
q3	22.77 22.84 23.95 24.02	7.02 7.69 7.85 10.92	23.95	7.85	−67.2%	FASTER	pandas→polars
q4	20.03 20.89 22.14 22.45	6.14 6.34 6.80 7.48	22.14	6.80	−69.3%	FASTER	pandas→polars
q5	101.10 102.67 102.81 103.57	7.37 8.10 8.61 11.86	102.81	8.61	−91.6%	FASTER	pandas→polars
q6	104.08 105.78 106.47 107.52	10.12 10.24 10.26 11.99	106.47	10.26	−90.4%	FASTER	pandas→polars
q7	39.14 39.21 39.57 40.19	10.19 10.54 10.60 11.78	39.57	10.60	−73.2%	FASTER	pandas→polars
q8	19.04 20.02 20.26 20.72	2.02 2.15 2.27 2.34	20.26	2.27	−88.8%	FASTER	pandas→polars
q9	34.23 37.68 38.48 39.03	14.71 15.27 15.72 16.25	38.48	15.72	−59.1%	FASTER	pandas→polars
nie	389.67 395.77 418.66 424.57	371.45 379.98 390.61 421.73	418.66	390.61	−6.7%	TIE	pandas→pandas

100k (107,434 nodes / 2,775,195 edges)

q	A per-slot (ms)	B per-slot (ms)	A med	B med	Δ	verdict	frames
q1	289.26 352.80 514.37 592.93	43.82 45.35 45.38 47.04	514.37	45.38	−91.2%	FASTER	pandas→polars
q2	434.52 483.38 536.71 604.67	53.75 54.94 57.25 57.51	536.71	57.25	−89.3%	FASTER	pandas→polars
q3	161.99 162.61 177.01 202.89	15.84 17.51 17.61 17.87	177.01	17.61	−90.1%	FASTER	pandas→polars
q4	154.23 154.73 165.14 167.18	12.01 12.17 13.04 13.63	165.14	13.04	−92.1%	FASTER	pandas→polars
q5	590.02 602.33 657.98 696.46	17.28 17.50 17.71 17.81	657.98	17.71	−97.3%	FASTER	pandas→polars
q6	609.75 614.65 654.05 655.09	23.51 23.94 28.03 28.59	654.05	28.03	−95.7%	FASTER	pandas→polars
q7	281.48 281.70 299.70 308.91	16.12 17.95 20.50 20.92	299.70	20.50	−93.2%	FASTER	pandas→polars
q8	161.20 168.27 172.52 174.41	5.17 5.33 6.44 6.68	172.52	6.44	−96.3%	FASTER	pandas→polars
q9	313.50 315.80 318.38 330.94	64.55 64.99 66.71 70.44	318.38	66.71	−79.0%	FASTER	pandas→polars
nie	5022 5101 5176 6662	5046 5102 5366 5450	5175.8	5365.6	+3.7%	TIE	pandas→pandas

Overlapping per-slot ranges are scored TIE, never a win. Every scored cell's per-slot ranges are disjoint by a wide margin, and canonical row values are identical on every cell of both scales.

What the untouched control arms say about the noise floor

Both scales also ran engine='pandas' and engine='polars' — paths this PR does not touch, so their true delta is zero and they measure the harness.

• 20k pandas: a uniform +1.1% to +6.5% tilt toward B on all nine, i.e. residual drift that favours A. The polars-auto wins are therefore conservative.
• 20k polars: nine TIEs and one spurious FASTER (q1, −11.9%).
• 100k pandas: flat (−1.8% to +4.2%), all TIE.
• 100k polars: eight TIEs and one spurious SLOWER (q5, +3.5%) with disjoint slot ranges.

Read that as the design's resolution floor: this 8-slot layout produces spurious non-overlapping verdicts up to about ±12% at 20k and ±5% at 100k. A claim in that band would not be safe from this harness. −59% to −97% is an order of magnitude outside it.

Explicitly hunting for what gets SLOWER

Nothing measurably regresses on the served shapes, at any of the four scales probed. The two structural ways this change can cost time were measured directly rather than assumed:

1. The declined-shape fallback tax. A shape the native engine declines pays the declined attempt and then the whole generic path. Isolated (engine='polars' timed until it raises):

graph	declined attempt	full routed call	tax
50 nodes	0.226 ms	60.1 ms	0.38%
1k nodes	0.223 ms	110.4 ms	0.20%
20k nodes	0.226 ms	952.4 ms	0.02%

A flat ~0.23 ms, independent of graph size — the decline is raised at planning, before any data is touched. On the lane's nie cell it is invisible (TIE at both scales).

2. Small-graph fixed cost. The native path could carry a larger constant than the bridge. It does not: at 50 / 1k / 20k nodes across four cheap shapes, arm B is faster or equal on every served cell (e.g. 50-node cheap_scan 12.02 → 1.16 ms; 50-node cheap_seed 21.66 → 1.17 ms).

VALUES, not just latency

A 45-shape matrix (node scans, typed/untyped filters, nulls, toLower, arithmetic, string concat, seeded 1-hop and 2-hop, undirected, five aggregate families, group-by, DISTINCT, ORDER BY/SKIP/LIMIT, multi-alias joins, OPTIONAL MATCH, variable-length, shortestPath, WITH, edge-property filters, whole-entity returns, empty results, and three native chain-API shapes) was run under AUTO on both builds, comparing row counts, canonical values, column dtypes, column order and row order.

Row values: 0 mismatches across all 45 shapes once integer/float representation is normalised, and 0 status changes. 44/45 routed natively; shortestPath took the fallback.

The differences that are real, and which the reviewer should rule on:

1. Frame type — pandas → polars everywhere. The declared contract change.
2. Nullable integers — 30.0 → 30. The old default upcast to float64 because pandas had no nullable int; polars keeps Int64. Values are equal numerically; the printed representation and the dtype both change. Same for nullable booleans (object → Boolean).
3. Column ORDER on the returned _edges — an aliased edge column moves from first to last (['r','s','d',…] → ['s','d',…,'r']) in 5 of 45 shapes. Column sets are identical.
4. A leaked internal column disappears — a -[*1..2]-> result carried __gfql_output_edge_hop__ in _edges under the old default and does not under the new one.

Two divergences that are NOT introduced here but that this PR makes the DEFAULT

These already exist between engine='polars' and engine='pandas' today. They are surfaced because a default-routing change is what promotes them from opt-in to universal. I am not proposing to decide these unilaterally.

query	old default	new default
RETURN sum(a.<string col>)	'abac' (pandas concatenates strings)	raises polars.exceptions.InvalidOperationError
RETURN avg(a.<string col>)	raises GFQLTypeError [type-mismatch]	silently returns null

min/max over strings agree on both engines.

Neither old behaviour is openCypher-correct (both should be type errors), but the directions differ in how much they matter:

• sum() — a working query becomes a hard failure, and the exception is a raw polars type leaking through the GFQL surface rather than a GFQLTypeError. That is an error-hygiene defect independent of the routing question.
• avg() — a query that used to raise a typed error now silently returns null. This is the direction that worries me: it is the silent-wrong-answer class, arriving by default.

My recommendation is that the avg()-on-string case be fixed in the polars engine (decline or raise GFQLTypeError) before this lands, and the sum() case at least be wrapped so a GFQLTypeError is raised instead of a leaked polars exception. Both are small and separable from this PR. Found by a 107-query differential sweep of AUTO-vs-AUTO; the other 99 were identical.

5. One extra OTel span per routed query. The guard recurses through the module-level gfql name, which is the @otel_traced("gfql.run")-decorated wrapper, so an AUTO-routed polars query now enters it twice (measured: 2 entries per user call on the routed path, 1 on the pandas path and 1 on explicit engine='polars'). Each user call therefore emits a nested duplicate gfql.run span and evaluates _gfql_otel_attrs twice. Results are unaffected. I have not changed this: the clean fix is to split the body into an undecorated _gfql_impl and have the guard call that, which is a mechanical but sizeable move on a file several other in-flight PRs touch, and a nested span arguably documents the routing decision. Flagging it as a reviewer call rather than deciding it here. (gfql.__wrapped__ would be a one-token fix but is untypeable without a banned cast.)

The policy hooks: a defect found, and fixed here

The original guard routed policy-carrying queries too. Measured, that was not safe:

• The native executor does not go through chain_impl, so it never emits postload / postchain. A policy that DENIES on postload blocked the query before and did not block it after. A governance hook that silently stops firing is worse than a slow query, and it would have applied to every user who never pinned an engine.
• The NIE fallback re-ran the query, firing preload / precompile / postcompile twice for one user call — a policy that counts or rate-limits would have double-counted.

policy is None is now part of the guard. The full hook trace and every deny/allow outcome across all six hooks is byte-identical to 233b64c8 on polars-frame graphs, pandas-frame graphs, and the declining shape. Explicit engine='polars' is unchanged and still carries the underlying gap — that gap is a separate, pre-existing bug worth its own issue; this PR only refuses to make it the default.

Regression lock-in

graphistry/pyg-bench#110 adds auto-engine-default-route to the gfql-perf-lockin lane.

It is a structural receipt, not a ladder, and that is the point: a growth ratio measures the fraction of a query proportional to the scaled dimension, and routing moves a constant, so a ladder would police the wrong direction and go green on a build that had silently reverted. The probe runs the named seeded pattern through the bare g.gfql(query) spelling on polars frames and asserts 2 = native polars frames out and zero pl.DataFrame.to_pandas calls; 1 = pandas frames out of a polars-frame graph (regression); 0 = invalid. Its positive and negative controls (engine='polars' must report 2, engine='pandas' must report 1) are part of the observation, so a probe that cannot discriminate cannot go green.

Validated both ways through the committed runner: 233b64c8 → 1 on every rung, verdict regression, exit 1; the same tree with this PR → 2 on every rung, exit 0. rows_returned {nodes: 33, edges: 32} on every rung of both.

Merge order: pyg-bench#110 is deliberately held until this lands. Against current pygraphistry master the probe correctly reports a regression — which is the proof that the gate works, but merging it first would leave gfql-perf-lockin red for a change that has not shipped, and a lane that is red for a legitimate reason is a lane people learn to ignore.

Tests

Positive: AUTO result frame-equal to explicit polars; edges-only graph (self._nodes is None is inside the guard's condition); both spellings of AUTO (enum and string); shortestPath NIEs when pinned and answers via the fallback.

Negative — for a routing change, what must not be routed matters as much: pandas frames untouched; mixed polars/pandas frames untouched; explicit engine='pandas' untouched; a denying postload policy still blocks; the compile/load hooks fire exactly once on a declining shape.

Verification

bin/lint.sh clean · bin/typecheck.sh clean (325 files) · graphistry/tests/compute/ — 123 failed on this branch and 123 failed on 233b64c8 in the same env, identical failure set (local umap/dask/env failures, unrelated and pre-existing); passes 6806 → 6815, the +9 being the tests added here.

🤖 Generated with Claude Code

https://claude.ai/code/session_015YsqAZQLbqjSDrYSFz2GoB

[lmeyerov]

Too early to enable by default