From 3894329237a850cec08ba7f5c8a6a119bfa61e1d Mon Sep 17 00:00:00 2001 From: Anton Date: Mon, 21 Sep 2026 17:37:32 +0200 Subject: [PATCH] fix(in): admit the typed kernel for text columns, which it never reached MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `in` gated its typed-kernel fast path on ray_vec_may_have_nulls for both operands. That predicate returns TRUE unconditionally for SYM and STR — their null is a payload value (id 0 / "") rather than an attribute bit, so attrs cannot prove a text column null-free. The gate was therefore never satisfiable for a text operand, and every `in` over a symbol or string column fell through to the generic per-row hashset probe. The kernel it could not reach carries a verdict-LUT written specifically for SYM: one byte load per row regardless of set size. It was dead code for the only type that path was built for. vec.h states the distinction where it defines the two predicates — may_have_nulls is "a row-kernel gate, not a reason to reject a text optimization", and "paths requiring null-free data use has_nulls below". This is an admission check, so it now asks ray_vec_has_nulls, which inspects the payload for text and is not on a per-row path. Measured, 351,393-row SYM column, 766 distinct values, -c 2: needles before after 1 4,820 us 200 us 24x 2 5,180 us 160 us 32x 10 8,320 us 160 us 52x 100 8,660 us 180 us 48x 766 8,440 us 180 us 47x 50,000 7,360 us 520 us 14x The reported symptom in #593 was the degenerate case — a one-element right side costing ~120x the equivalent `(== col x)`. That gap is now ~5x, but the fix is not a small-needle special case: it lands on every text `in`, which is why the whole column of numbers moves. Null semantics are unchanged, and that is the risk the old gate existed to manage: the kernel treats a null as matching nothing, while this path is null-equals-null. Admitting only exactly-null-free operands keeps the two from being conflated — a null on EITHER side still falls through. All eight null shapes were compared against the pre-fix binary and are identical; they are now pinned in test/rfl/collection/in.rfl, which had no null coverage at all, together with the null-free shapes that now reach the kernel and the degenerate single-needle case. ClickBench (10M rows, 43 queries x 3, splayed): sum of per-query hot times 5690.7 ms -> 5664.6 ms, no query beyond +4%. The exact check costs one pass over each operand and does not show. This site is the only one that had the defect. Every other admission gate reading may_have_nulls either excludes text by an explicit type guard below it (ray_sorted_range_rowsel takes numeric and temporal columns only; str_scalar_int's switch accepts no text type), handles text explicitly in the same condition (fills_vec_eager returns text unchanged by design), or reads the predicate in the direction where an always-true answer is the safe one. Checked rather than assumed, after an earlier draft of this message claimed the opposite. Refs #593 --- src/ops/collection.c | 17 +++++++++++++++-- test/rfl/collection/in.rfl | 39 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 54 insertions(+), 2 deletions(-) diff --git a/src/ops/collection.c b/src/ops/collection.c index b9ac009d8..6b20c4b24 100644 --- a/src/ops/collection.c +++ b/src/ops/collection.c @@ -1374,8 +1374,21 @@ ray_t* ray_in_fn(ray_t* val, ray_t* vec) { * the WHERE kernel uses null-matches-nothing semantics — the * two must not be conflated. NULL result = unsupported shape * (STR etc.): fall through to the hashset probe below. */ - if (!ray_vec_may_have_nulls(val) && - !ray_vec_may_have_nulls(vec)) { + /* Admission check, so it uses the EXACT ray_vec_has_nulls rather + * than the conservative row-kernel gate. ray_vec_may_have_nulls + * returns true unconditionally for SYM and STR — their null is a + * payload value, not an attribute bit — so gating this on it made + * the branch unreachable for every text column, including the SYM + * verdict-LUT the kernel carries specifically for them. Every + * `in` over a symbol column fell through to the generic per-row + * hashset probe: ~4.8 ms against ~40 us for the equivalent `==` + * on a 351k-row column (#593). vec.h says as much where it + * defines the two — "paths requiring null-free data use has_nulls + * below". The exact check costs one pass and is not on a per-row + * path; a null-bearing operand still falls through, because the + * kernel's null-matches-nothing semantics differ from this path's + * null-equals-null. */ + if (!ray_vec_has_nulls(val) && !ray_vec_has_nulls(vec)) { ray_t* fast = ray_in_vec_exec(val, vec, false); if (fast) return fast; } diff --git a/test/rfl/collection/in.rfl b/test/rfl/collection/in.rfl index fbead052c..026417e41 100644 --- a/test/rfl/collection/in.rfl +++ b/test/rfl/collection/in.rfl @@ -84,3 +84,42 @@ (in [1h 2h 3h] [3h 2h 1h]) -- [true true true] ;; Test with empty arrays (in (list) [1h 2h]) -- (list) + +;; ========== TEXT (SYM/STR) NULL SEMANTICS (#593) ========== +;; `in` admits the typed verdict-LUT kernel only when BOTH sides are exactly +;; null-free, because that kernel uses null-matches-nothing semantics while +;; this path is null-equals-null. The gate used to ask +;; ray_vec_may_have_nulls, which is unconditionally true for SYM and STR +;; (their null is a payload value, not an attribute bit) — so the kernel was +;; unreachable for text columns and every symbol `in` fell through to the +;; generic per-row hashset probe. These pin the semantics on both sides of +;; that gate so the admission check cannot drift back. + +;; null in the COLUMN, null needle — null equals null +(in (as 'SYM ["a" "" "b"]) (as 'SYM [""])) -- [false true false] +(in ["a" "" "b"] [""]) -- [false true false] + +;; null in the column, non-null needle — the null matches nothing else +(in (as 'SYM ["a" "" "b"]) (as 'SYM ["a"])) -- [true false false] + +;; null only in the NEEDLES: the column is null-free but the gate must still +;; reject, since a null needle needs null-equals-null against nothing +(in (as 'SYM ["a" "b"]) (as 'SYM ["" "a"])) -- [true false] + +;; nulls on both sides +(in (as 'SYM ["a" ""]) (as 'SYM ["" "z"])) -- [false true] + +;; null-free text: this is the shape that now reaches the kernel +(in (as 'SYM ["a" "b" "c"]) (as 'SYM ["b" "c"])) -- [false true true] +(in ["a" "b" "c"] ["b" "c"]) -- [false true true] + +;; empty needle set over a null-free symbol column +(in (as 'SYM ["a" "b"]) (as 'SYM [])) -- [false false] + +;; a single needle is the degenerate case the issue reported: same answer as +;; the equality it should cost the same as +(in (as 'SYM ["a" "b" "a"]) (as 'SYM ["a"])) -- [true false true] +(== (as 'SYM ["a" "b" "a"]) 'a) -- [true false true] + +;; duplicate needles must not double-count or change the verdict +(in (as 'SYM ["a" "b"]) (as 'SYM ["a" "a" "a"])) -- [true false]