String expressions on the compute engine, and utf8() to query parity - #686
Merged
Conversation
miniexpr can now return strings, but the output container is allocated before
evaluation, so it has to be sized from miniexpr's own width inference. Sizing
it from numpy's result_type made the kernel silently truncate: `<U16` + 'XYZ'
produced a `<U16` container and dropped the suffix.
- blosc2_ext: declare me_get_dtype/me_get_itemsize and add me_output_dtype(),
which probe-compiles the expression with ME_AUTO and reports the inferred
result dtype. Returns None when miniexpr cannot type the expression, so the
caller keeps its numpy fallback -- which is what bytes ('S') still does.
- LazyExpr.dtype consults it whenever a string operand is involved, ahead of
the dtype cached during expression building.
Also fixes NDArray.__radd__, which delegated to __add__ and so silently
reversed the operands. Harmless while `+` only meant addition; wrong now that
it also means concatenation: "prefix" + arr produced arr + "prefix".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The DSL grammar has no attribute syntax, no `in` operator and no tuple
unpacking, so a pandas UDF written in ordinary Python does not parse. That
defeats the point of `df.apply(f, axis=1, engine=blosc2.jit)`, which is to swap
in the engine without rewriting the function.
_StringSyntaxRewriter handles it as an AST pass, alongside the existing
_NumpyAttrCallRewriter, rather than as C parser work:
s.lower() -> lower(s)
x in s / x not in s -> contains(s, x) / not contains(s, x)
a, b = s.split(sep, 1) -> a = split_part(s, sep, 0)
b = split_part(s, sep, 1)
The unpack deliberately emits two statements on separate lines rather than a
';'-joined one: miniexpr now rejects that form outright, and silently dropped
the second statement before. Only the maxsplit=1 shape is handled, which is
what tuple unpacking can consume.
LazyUDF also asks miniexpr for the output dtype of a string-valued kernel, for
the same reason LazyExpr.dtype does: the container is allocated up front and
nothing on the Python side can predict a concat or case-mapping width.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
lazyudf() resolved a DSL kernel's dtype by np.result_type over the input dtypes before LazyUDF got a chance to ask miniexpr, so a string-returning kernel was allocated at the operand width and miniexpr wrote past it. - lazyudf(): ask miniexpr first for string kernels, promote as before otherwise. - _set_pref_expr(): the container is allocated before the compile, so verify the width miniexpr infers matches it instead of overrunning the block. - tests: pass strict_miniexpr=True throughout (these all passed on the numpy fallback before), and add the pandas-3 blog kernel as a DSL kernel. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
miniexpr grew ME_BYTES, so `S` operands no longer need the numpy fallback. - `_me_dtype_from_numpy_dtype`: kind "S" -> ME_BYTES. - both `v.dtype.num == 19` itemsize gates now accept 18 (NPY_STRING); only one of the two was reachable for `U`, and both are for `S`. - `me_output_dtype` reports an `Sn` dtype back, and the string-dtype gates in lazyexpr test `kind in "US"`. - the DSL validator accepts `bytes` constants, so `b"x" in col` compiles. Verified against np.strings: concat width, ASCII-only case mapping (upper on `S8` stays `S8`), predicates, and a bytes DSL kernel. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`df.apply(f, axis=1, engine=blosc2.jit)` could not run any kernel combining
`row["colname"]` with control flow — not even a numeric one. Tracing
evaluated the `if` over a whole column ("truth value ... is ambiguous") and
the DSL parser rejected the subscript first.
_RowSubscriptRewriter (alongside the other AST rewrites) turns
`def f(row): ... row["a"] ...` into `def f(a): ... a ...` when every mention
of the row parameter is `param[<string literal>]`; anything else still bails
and is rejected as before. DSLKernel keeps the original labels, since they
need not be identifiers, and _jit_dsl_wrapper pulls those columns out of the
single row-proxy argument.
String columns reach this route too: the whole-frame numeric gate is skipped
for it (it reads one column at a time), `_miniexpr_eligible_operand` accepts
"U"/"S", and `np.asarray` is bypassed -- on a pandas 3 `str` column it yields
an object array of PyObject pointers. With that, the pandas-3 "format room
info" kernel runs unmodified and byte-identically, reporting engine=miniexpr.
Nulls in a string column are rejected rather than substituted: a row-wise
kernel over a null raises in pandas too, so quietly making it "" would invent
a value pandas never produces. The error names the column and the fix.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
c-blosc2 caps a typesize above BLOSC_MAX_TYPESIZE (255) to 1 in the chunk
header so its split machinery keeps working ("treat buffer as an 1-byte
stream", blosc2.c). aux_miniexpr() asked blosc2_getitem_ctx() for the
operand block in *element* units, which the chunk then interpreted as
bytes: block 0 came back with only its first few elements populated and
every later block was the untouched malloc'd buffer. Results were wrong
and non-deterministic -- `arr == "hello"` over 1200 rows of <U64 matched
1 row instead of 400 -- with no error raised anywhere.
Convert the request through bytes using the typesize the chunk header
actually records (blosc1_cbuffer_metainfo). Identical to the previous
arithmetic whenever the typesize is not capped.
<U64 is 256 bytes, the first fixed-width string that trips this; the
Phase 1/2 suites only ever used widths up to <U32.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Lifts the NotImplementedError on `t.where("name == 'x'")` and friends for
variable-length utf8 columns. Only the operator form `t[t.name == "x"]`
worked before.
A utf8 column cannot be an expression operand: its offsets and data live
in separate NDArrays with independent chunk grids, so the prefilter
contract does not apply. So drive it instead -- _utf8_span_eval() walks
the column in row spans, materializes each span to a fixed-width <Un
array (width rounded up to a power of two, so a column costs a handful of
compilations rather than one per span) and hands that to miniexpr.
Span operands are passed as blosc2 arrays, not NumPy ones: the NumPy
route evaluates through slices_eval, which never reaches miniexpr, so the
string kernels would have been bypassed for correct-looking results.
_utf8_span_eval(strict=True) asserts that.
Nulls (3c): materialized to "" so no C kernel ever sees a sentinel, and
nullity re-applied afterwards -- a boolean result is forced False, which
is what the operator form and SQL WHERE semantics both give. Tests pin
the two forms against each other, including a null against the sentinel
string itself.
Span sizing is bounded by _UTF8_EXPR_BUDGET as well as row count: the
width comes from the *longest* value in the span, so one 4 KB row among
short ones would otherwise materialize 65536 x 4096 x 4 = 1 GiB. The
byte lengths come from the offsets, so no span is read twice to size it.
3a: blosc2.utf8_array(seq, spec=None) builds a Utf8Array from an iterable;
Utf8Array is exported too. Construction was previously Utf8Array(spec) +
.extend() + .flush(), not exported at all. The name shadows the internal
module of the same name, so two monkeypatch call sites in the tests now go
through sys.modules.
Still unsupported, and still raising clearly: nested (dotted) utf8 leaves,
and utf8 columns in computed-column expressions (those need a LazyExpr,
not a materialized value).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
They land with the span-loop driver; create_index() is the one remaining utf8 limitation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Exporting blosc2.utf8_array() as the public Utf8Array constructor shadowed the module of the same name. `from blosc2.utf8_array import X` still resolved (the import machinery finds the submodule), but attribute-path lookups landed on the function, which broke two monkeypatch call sites in the tests and would have confused anyone reading `blosc2.utf8_array` in a traceback. The module was always internal -- every reference to it is inside blosc2 or its tests -- so the leading underscore states that and frees the plain name for the constructor. The monkeypatch sites go back to the plain attribute-path form. No public API change: blosc2.utf8_array (function), blosc2.Utf8Array and `from blosc2 import ...` are unaffected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The plan gated Phase 4 (a native ME_UTF8 input dtype in miniexpr) on
measurement, because its competition is not the span driver's decode but
the raw-byte comparison python-blosc2 already has. Measured, and the
gate says do not build it:
1M rows, <U32 values, t.where("name == 'x'")
decode (StringDType read) 27.1 ms
+ astype(<U32) 115.6 ms
blosc2.asarray(span) 45.7 ms feeding miniexpr
miniexpr eval 19.6 ms
------------------------------------
full expression path 262.9 ms
operator t[t.name == "x"] 53.0 ms raw bytes, no decode
miniexpr is 19.6 ms of 263. A native ME_UTF8 would delete the decode and
the astype and land at roughly the operator form's 53 ms -- which is what
Column._utf8_scalar_mask already does today, in NumPy, for free. So the
1-1.5 weeks of threading a new dtype through the evaluator buys what a
rewrite pass buys.
So do the rewrite pass instead: `utf8col <cmp> 'literal'` terms (both
operand orders, all six comparisons) are answered by the raw-byte scan and
substituted into the expression as boolean operands, mirroring the
_rewrite_dictionary_predicates pass that already sits next to it. A utf8
name drops out only when every one of its occurrences was rewritten, so
startswith/contains/upper still route to the span driver, and a mixed
expression rewrites the half it can.
1M rows short 156.5 -> 28.2 ms (5.5x)
1M rows medium 267.5 -> 56.3 ms (4.8x)
200k rows long 97.7 -> 16.0 ms (6.1x)
The expression form now matches the operator form, which was Phase 4's
target. Tests assert the route taken, not just the answer, since
correctness alone cannot tell the two apart.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`create_index()` on a string column silently made every query on it match
nothing. No error, no warning -- adding an index, an optimization, changed
the answer to zero rows:
@DataClass
class Row:
name: str # -> string(max_length=32) -> <U32
x: int
len(t.where("name == 'c07'")) # 250
t.create_index(col_name="name")
len(t.where("name == 'c07'")) # 0
Same root cause as cbcb15b, reached through a second caller. A segment
summary is a `(min, max, flags)` record, so a <Un column makes it 8n+1
bytes: 257 for max_length=32, which is the *default* width for a plain
`str` annotation. Above BLOSC_MAX_TYPESIZE (255) c-blosc2 records the
chunk typesize as 1, and get_1d_span_numpy() asked blosc2_getitem_ctx()
in element units -- so the sidecar decoded to a byte range's worth of
garbage plus uninitialised tail, and _candidate_units_from_summary()
pruned every candidate away.
The boundary is exactly 8*max_length + 1 > 255: max_length 31 works, 32
does not. summary/bucket/partial/full were all affected; opsi survived
because it does not read segment summaries.
Convert the request through bytes using the typesize the chunk header
records, as in cbcb15b, and raise on a short read instead of leaving the
tail of the destination uninitialised -- that silence is what made this
cost a full investigation rather than showing up as an error.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This defect surfaced three times in one session through three different callers, so give it a single home instead of a fourth copy. New getitem_span() converts an element-addressed span through bytes using the typesize the chunk header records, and every blosc2_getitem_ctx() call now goes through it: the miniexpr prefilter (cbcb15b), the index sidecar reader (0546293), and the matmul prefilter -- the last unreachable today at typesize <= 8, routed through the helper so it does not become a latent trap. Callers now compare the decoded byte count against what they asked for, so a short read raises instead of leaving a buffer partly uninitialised. The sweep also turned up a fourth instance, this one upstream: blosc2_schunk_get_slice_buffer() derives the getitem for a partially covered chunk by dividing byte offsets by schunk->typesize, which the header contradicts above 255. Over 153 slice shapes at typesize 256, 150 raised "Error while getting the slice" and 3 -- the single-element ones -- returned the wrong bytes with no error. Reachable from ordinary data: an <U64 NDArray is a 256-byte typesize, and arr.schunk[1:4] hit it. Route around it in SChunk.get_slice() by decompressing the covered chunks whole and cutting the range out, which never calls getitem. All 153 shapes now correct. Worth reporting upstream to c-blosc2 as well; the workaround is local and cheap enough to keep regardless. Audited the remaining typesize arithmetic in blosc2_ext.pyx: everything else divides our own metadata (chunksize // typesize and friends) and never meets a chunk header, so this closes the family. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
So the workaround can be dropped when the upstream fix lands in the pinned c-blosc2, rather than outliving the bug it routes around. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Blosc/c-blosc2#796 is fixed: blosc2_schunk_get_slice_buffer() and the single-coordinate path of blosc2_schunk_get_sparse_buffer() now convert through the typesize chunks actually carry, so a partially covered chunk at typesize > BLOSC_MAX_TYPESIZE addresses the right range. Upstream also made partial getitem decodes return BLOSC2_ERROR_DATA rather than looking like success, which is how every downstream instance of this family stayed silent. SChunk.get_slice() goes back to a plain super() call. The three regression tests added with the workaround stay and now exercise the C path -- without the fix, 150 of 153 slice shapes at typesize 256 raise and 3 return wrong bytes, so their passing is what verifies the fix is live in the linked c-blosc2. getitem_span() in blosc2_ext.pyx stays: blosc2_getitem_ctx() still counts in the header typesize by design, so the miniexpr prefilter, the index sidecar reader and the matmul prefilter still need the element-to-byte conversion. The fix is post-3.2.3 and unreleased, so BLOSC2_MIN_VERSION (3.2.1) is now too low for USE_SYSTEM_BLOSC2 builds -- a system 3.2.3 compiles fine and silently reinstates the bug. Bump it to 3.2.4 once that ships. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
blosc2_getitem_ctx() counts in the typesize the chunk records, which c-blosc2 caps to 1 above BLOSC_MAX_TYPESIZE, so its unit changes silently with the data. Upstream now offers blosc2_getitem_bytes_ctx(), which counts in bytes at any typesize (Blosc/c-blosc2 bc074b22, follow-up to #796). Every partial read here moves to it, and the cap rule goes back to living only inside c-blosc2. The three sites are the ones getitem_span() was written for: the miniexpr prefilter, where a <U32 operand is a 129-byte typesize; the index sidecar reader, where a <U32 summary is a 257-byte record; and the matmul prefilter, unreachable at typesize <= 8 today but asked in the unambiguous unit so it cannot become one. Nothing else in the tree reads a sub-range of a chunk -- the append path and the mask scan in ctable_indexing want whole chunks by nature. getitem_span() itself is gone rather than rewritten. Once the cap rule moved out, it was a multiply undoing a division its callers had done a line earlier: each site already held the byte count and already compared rc against it, so passing bytes reads more directly than passing an element count plus a typesize. That also left matmul's blocknitems[] write-only. The API requires start and nbytes to be multiples of the stored typesize. Every offset here is derived from the real typesize, so this is exact below the cap and vacuous above it. Note that BLOSC2_MIN_VERSION (3.2.1) is now too low: this is a build-time dependency, so a system c-blosc2 without the new entry point passes the CMake gate and fails at compile. Bump it to 3.2.4 once that ships. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The contagion rule from the design (a string-returning expression with a utf8 operand produces a utf8 result) had no implementation on the expression side: _utf8_span_eval() accumulated every result into one physical-length NumPy array, widening it as later spans returned wider <Un values. For a string result that is miniexpr's compile-time bound paid on every row -- and lower() reserves a 2x case-expansion factor at 4 bytes per codepoint on top. String results now extend a Utf8Array span by span instead, so only one span's <Un block is ever live and the stored result costs each row its own UTF-8 length. Bool and numeric results are unchanged. Nulls keep the documented policy: the sentinel is restored into the values and the result spec carries it, so the Utf8Array is nullable exactly when its source was. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds me_eval_varlen()/me_varlen_data_bound(). Nothing in blosc2 calls them yet, so this is a no-op for the built extension -- the static linker never pulls the new object in -- but the pin has to move before the blosc2 side can use it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Wires miniexpr's me_eval_varlen() through to Python:
- blosc2_ext.eval_varlen() compiles and evaluates into Arrow int64 offsets plus
a UTF-8 blob, releasing the GIL around the evaluation;
- Utf8Array.extend_encoded() appends offsets+bytes in bulk, with no decode to
str on the way in (_rewrite_from's tail is now shared as _write_encoded);
- blosc2.compute_varlen() runs a LazyExpr or a DSL-backed LazyUDF in row spans
across a thread pool into a Utf8Array. Varlen output has no fixed
per-element stride, so the prefilter cannot carry it and the spans are where
the parallelism has to come from.
This was built to close the Chicago Taxi benchmark's gap against DuckDB, and
the measurement says the gap was never there. 1M rows, transform, one engine
per process:
fixed-width <U66 264 B/row -> 0.81 MB stored, 133 ms
varlen 34 B/row -> 1.14 MB stored, 149 ms
The blob lands on DuckDB's 35.9 B/row exactly and still loses on both axes.
blosc2 stores results compressed, and the fixed-width form's NUL padding
compresses to nearly nothing while a dense UTF-8 blob has nothing left to
squeeze. The 404 B/row that motivated this is the uncompressed result, which
blosc2 never stored. On time, break-even is the ceiling: eval 290 ms + accum
70 ms serial, against a prefilter that runs in blosc2's C thread pool fused
with compression.
Kept as a representation feature, not a performance one -- it is the only route
from an expression to an Arrow varlen result, which a Utf8Array-typed computed
column will need. The docstring says so. In the benchmark it is opt-in via
--engines "blosc2,blosc2 (varlen)"; running an engine second costs ~40% on this
machine, which is why the two are compared one per process.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…sions" This reverts commit f6b0643.
string-ops.py was untracked before the compute_varlen commit, so that commit was what first added it and the revert removed it wholesale. Back without the varlen engine, the Utf8Array footprint branch, or the StringDType digest branch -- none of which have a caller now. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
upper()/lower() no longer reserve a 3x/2x case-expansion bound on <U, which matches what numpy does. String results narrow accordingly: the Chicago Taxi kernel output goes from <U101 to <U54, halving the uncompressed result. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A <U result was being compressed 3.2x worse than the identical bytes written
by asarray(): 2.01 MB vs 0.63 at 1M rows, with byte-identical chunk contents,
identical headers (flags, typesize, nbytes, blocksize, filter pipeline) and
identical geometry. The only difference was filters_meta, which is SHUFFLE's
element width: [0,...,0] on the expression path against [0,...,4] on asarray's.
Left alone, the container picks 4 for <U -- the UCS4 code unit -- so shuffle
groups the (mostly zero) high bytes of every codepoint together. CParams
defaults filters_meta to all zeros, meaning "shuffle by the whole item", so
merely constructing a CParams to set something unrelated scatters characters
across the slot instead. Three sites did that:
- fast_eval() passed CParams() whenever the caller gave none. It now passes
nothing, letting uninit() choose;
- the reduction path needs NEVER_SPLIT, so it keeps its CParams and restores
the width through the new _restore_code_unit_shuffle() helper;
- LazyUDF.compute() always set aux_kwargs["cparams"], even when the merged
dict was empty -- and cparams={} still materializes CParams' defaults. It
now only sets it when something was actually requested.
1M rows of the Chicago Taxi string benchmark:
transform 179.6 ms / 2.0 MB -> 124.3 ms / 0.6 MB
kernel 286.2 ms / 2.0 MB -> 232.1 ms / 0.6 MB
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…rands
The result container inherits the operands' block shape in *rows*, and a string
result is much wider per row than its operands (<U54 against <U36), so a row
count tuned for the operands gives a far bigger byte-block for the result. At
the pinned 8192 rows the <U54 result blocks were 1.7 MB -- well out of cache --
and every task paid for it.
512 rows puts them at ~108 KB, which is what asarray() picks for itself when
left alone. The pin itself has to stay: with auto blocks the two operands get
different grids and the expression falls off the miniexpr fast path entirely
(strict_miniexpr catches it).
1M rows, default cparams:
filter 28.1 -> 15.6 ms transform 124.1 -> 79.7 ms
kernel 231.6 -> 152.3 ms
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ZSTD-5 spends most of the blosc2 time on the string tasks for a ratio nothing
here needs -- 68 MB against DuckDB's 842 either way. LZ4-5 is ~1.6x faster to
write for ~3.6x more stored bytes, still 14x below the Arrow-backed engines.
Full 24.3M-row table, against the ZSTD-5 numbers it replaces:
filter 469 -> 297 ms transform 2.11 -> 1.22 s
kernel 4.03 -> 3.09 s result 18 -> 68 MB
That is fastest of all five engines on transform (1.62x DuckDB), ahead of
DuckDB on filter, and within 1.04x on kernel. Compression also stops costing
anything: the compressed run now beats the clevel=0 one on both transform and
kernel, since a compressed block is less memory traffic than a 5.8 GB result.
filters_meta has to be written out explicitly. It is SHUFFLE's element width,
and a <U container picks 4 for itself -- the UCS4 code unit -- but constructing
a CParams for any reason defaults it to 0, "shuffle by the whole item". Same
transform: 6.55 MB / 77.6 ms against 2.72 MB / 48.9 ms.
blosc2 (raw) carries the same filter pipeline now, so clevel really is the only
variable between the two blosc2 rows.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CI's NumPy 1.26 job failed 17 tests in test_string_output.py. Every one was a *reference* computation, not the blosc2 call under test: `np.strings` and the `+` ufunc loop for U/S arrays are both NumPy 2.0 additions, so `expected = np.strings.upper(names)` and `expected = names + other` raised while the miniexpr side they were being compared against ran fine. `np.char` is the equivalent that exists in both, and is an exact substitute: same values, same result dtypes, and the same full case mapping (straße -> STRASSE) the case-expansion test is there to pin down. Verified against np.strings on NumPy 2.4.6 before swapping. Fixed alongside, and not yet seen by CI: test_ctable_indexing.py builds a utf8 dataclass at module scope, and blosc2.utf8() raises on NumPy < 2.0 (it calls string_dtype() to fail early), so the whole module would have failed to *collect* rather than merely failing a few tests. The utf8 half of the three rank-index parametrizations now carries a skipif; the dictionary half still runs, since it needs no StringDType. Checked by running the suite with numpy.dtypes.StringDType deleted: no collection errors and no failures. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
t.where("trip.name == 'x'") raised NotImplementedError on a utf8 leaf,
while the same query on a <Un, bytes() or dictionary() leaf worked --
utf8 was the only flavour whose dotted names could not be queried at all.
Dotted names are aliased to safe identifiers by _rewrite_nested_expression,
but that only rewrites names present in the operand namespace, and utf8
columns are excluded from it: a variable-length column cannot be an
expression operand, which is why the span driver exists. So a dotted utf8
leaf reached blosc2.lazyexpr still spelled with dots, where it is not a
parseable identifier.
Alias them in _lazyexpr_over_cols instead, sharing one _alias_dotted helper
with the nested rewrite. _rewrite_utf8_predicates and _utf8_span_eval take
the alias -> column map, since they must reach storage and the null sentinel
by column name while matching the alias in the expression. Covers both the
raw-byte scalar-mask route and the span driver.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The plan predates the conversion pair. What shipped instead is the opposite rule -- utf8 stores and filters, fixed-width computes -- so computed columns, DSL kernels and the bare-array lift are withdrawn rather than superseded: they would buy an API indistinguishable from <U at 3-5x the cost, in exchange for a StringDType-in-schema round trip whose failure mode is a table that will not reopen. G5 is marked shipped and pulled out of that group; its diagnosis here was wrong about the mechanism, so the correction is recorded next to it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The matrix filed nested (dotted) leaves under Compute, which read as another casualty of the G2/G3 decision. It was not: a dotted utf8 leaf could not be filtered either, which made "utf8 stores and filters" false for nested columns. Moved to the Query block, now green, with the mechanism and the decision written up in a new section. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A row's alphabetical rank orders exactly as its decoded value does -- the basis of the FULL rank index -- so _build_lex_keys can hand lexsort an int32 key and skip both the decode and lexsort's string comparisons. The code -> rank map is extracted as _dict_code_to_rank so the sort and the index builder cannot drift apart. 200k rows, cardinality 5000: key construction 106.2 -> 21.4 ms, sort_by(view=True) 246.7 -> 156.9 ms, descending 273.5 -> 156.2 ms (an object key could not be negated, so descending went through a double argsort that an int32 key does not need). _sorted_small_copy_from_live_positions held a second copy of the key builder with the same decode, a narrower dtype list (no StringDType keys) and a KeyError for computed sort columns, and it re-read the column after already gathering the codes. It now calls _build_lex_keys with the arrays it gathered, which removes the duplicate. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three findings, all real: - @blosc2.jit forwarded every decorator kwarg to blosc2.asarray() when the traced function returned a NumPy array, so combining a storage kwarg with an execution-tuning one -- @blosc2.jit(jit=False, cparams=...) -- raised instead of returning an NDArray. Only storage kwargs go there now; the function has already run, so there is nothing left to tune. The sibling compute() call keeps taking all of them on purpose: compute() names fp_accuracy while lazyudf() does not, so stripping them would drop it. - The mandelbrot benchmark's header still said any non-None jit() kwarg flips the return container. Only storage kwargs do. - _fill_me_udata leaked np_data/np_typesizes on every allocation-failure exit after the one that allocates them. All the exits now go through one _free_me_udata_tables() helper, so a table added later cannot be missed by some of them again. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
346 of 7164 defs were over 50 chars, every one of them a test -- the longest shipped name is 46. The names had become failure messages, spelling out the whole test matrix in the identifier. Trimmed the redundancy first (a prefix echoing the module, "column" for "col", "expression" for "expr"), then the trailing clause where it only restated the assertion. Where a rename dropped something the body does not otherwise say -- "when threads forced", "and honors explicit cparams" -- it moved into a one-line docstring rather than vanishing. Also collapsed the four test_batcharray_guess_items_per_block_uses_* tests into one parametrized test: they were a parametrize table written out longhand, four functions differing only in clevel and payload size. Test count is unchanged (7922 collected before and after). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR substantially expands Blosc2’s string capabilities across both the NDArray compute engine (via miniexpr) and CTable query/indexing, while also tightening correctness around varlen/dictionary comparisons and wide-typesize slicing. It also adds extensive tests and documentation to lock in the new behaviors and performance-oriented design choices (e.g., rank-based indexing for utf8/dictionary and “utf8 stores/filters; fixed-width computes”).
Changes:
- Enable/extend string operations (concat/case/strip/replace/substr/split_part, DSL method syntax + tuple-unpacking lowering) on the compute engine for fixed-width
U/Sarrays, and add regression tests for wide string typesizes and string-valued outputs. - Bring
utf8()columns to parity on the query surface (where/sort_by/group_by/create_index) via rank-based indexing, plus improved planner behavior for bucket indexes. - Improve pandas
engine=blosc2.jitaxis=1 handling (row subscript rewriting + string column support) and fix multiple “silent-wrong” behaviors (notably elementwise comparisons for varlen/dictionary containers).
Reviewed changes
Copilot reviewed 85 out of 87 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_tree_store.py | Renames tests for clearer intent around discovery/reopen behaviors. |
| tests/test_schunk_get_slice.py | Adds coverage for wide typesize slicing correctness and out= buffer handling. |
| tests/test_random.py | Test renames for clarity/consistency. |
| tests/test_python_blosc.py | Test rename for clarity. |
| tests/test_proxy_schunk.py | Test rename for readability. |
| tests/test_pandas_udf_engine.py | Updates/extends pandas UDF engine tests; adds control-flow row-kernel coverage. |
| tests/test_objectarray.py | Test renames; adds regression test for elementwise varlen-scalar comparisons. |
| tests/test_locking.py | Test rename for readability. |
| tests/test_list_array.py | Test renames for clarity. |
| tests/test_group_reduce.py | Test renames for readability. |
| tests/test_dict_store.py | Test renames for discovery + extensionless behavior. |
| tests/test_batch_array.py | Refactors blocksize budget tests into parametrized form; test renames. |
| tests/test_b2view_model.py | Test renames for clarity. |
| tests/ndarray/test_string_output.py | New test module for miniexpr string outputs (values + width inference) and wide-string regressions. |
| tests/ndarray/test_slice.py | Test renames for aligned-chunk detection coverage. |
| tests/ndarray/test_proxy.py | Test rename for readability. |
| tests/ndarray/test_ndarray.py | Test renames; adds/clarifies dtype-change copy semantics tests. |
| tests/ndarray/test_linalg.py | Test renames; clarifies BLAS-thread-limiting behaviors. |
| tests/ndarray/test_lazyexpr.py | Test renames for miniexpr routing/fallback behaviors. |
| tests/ndarray/test_jit.py | Test renames; adds regression for mixed kwarg forwarding behavior. |
| tests/ndarray/test_jit_dsl_dispatch.py | Test renames; clarifies DSL dispatch expectations. |
| tests/ndarray/test_getitem.py | Test renames; adds docstring clarifying structured sparse take path. |
| tests/ndarray/test_dsl_kernels.py | Broad test renaming + small docstring additions for DSL kernel behaviors. |
| tests/ctable/test_where_expressions.py | Test renames for readability. |
| tests/ctable/test_vlstring_vlbytes.py | Adds Column.assign() tests for varlen scalar columns; test renames. |
| tests/ctable/test_varlen_columns.py | Test rename for readability. |
| tests/ctable/test_table_persistency.py | Test rename for readability. |
| tests/ctable/test_sort_by.py | Test renames for clarity. |
| tests/ctable/test_schema_validation.py | Test renames for clearer intent. |
| tests/ctable/test_schema_specs.py | Test rename for readability. |
| tests/ctable/test_schema_mutations.py | Adds add_column(values=...) coverage; renames tests for clarity. |
| tests/ctable/test_parquet_interop.py | Test renames; clarifies CLI/interop behaviors via docstrings. |
| tests/ctable/test_object_spec.py | Test rename for clarity. |
| tests/ctable/test_nullable.py | Test renames; clarifies null-policy semantics. |
| tests/ctable/test_null_expressions.py | Test renames; clarifies null semantics in derived expressions. |
| tests/ctable/test_nested_metadata_root.py | Test renames for readability. |
| tests/ctable/test_nested_access_storage.py | Test renames for readability. |
| tests/ctable/test_groupby.py | Test renames; expands coverage for dict-key ordering/positions and deterministic sort behavior. |
| tests/ctable/test_getitem_access.py | Test renames for readability. |
| tests/ctable/test_dictionary_column.py | Updates UTF8Spec naming; adds tests for dictionary decode caching, comparisons, and rank-based sort keys. |
| tests/ctable/test_ctable_take.py | Test renames for clarity. |
| tests/ctable/test_ctable_ndarray_columns.py | Test renames; adds lifecycle docstring for generated columns. |
| tests/ctable/test_ctable_dataclass_schema.py | Test rename for readability. |
| tests/ctable/test_ctable_computed_cols.py | Test renames for clarity. |
| tests/ctable/test_csv_interop.py | Test rename for readability. |
| tests/ctable/test_column.py | Test renames; clarifies fast-path behaviors for setitem and info. |
| tests/ctable/test_column_slice_fastpath.py | Test rename for readability. |
| tests/ctable/test_column_ndarray_like.py | Test rename for readability. |
| tests/ctable/test_arrow_interop.py | Test rename for readability. |
| tests/b2view/test_sort.py | Test rename for clarity. |
| tests/b2view/test_plot_model.py | Test renames; clarifies semantics for windowed plotting. |
| tests/b2view/test_group.py | Test renames for readability. |
| tests/b2view/test_cli.py | Test renames for readability. |
| src/blosc2/schema.py | Renames Utf8Spec→UTF8Spec (alias kept); updates utf8() docs and rejects NUL-only sentinel. |
| src/blosc2/schema_compiler.py | Updates spec mapping/type checks to UTF8Spec. |
| src/blosc2/scalar_array.py | Adds set_all() and elementwise __eq__/__ne__ for varlen scalar storage; keeps identity hashing. |
| src/blosc2/proxy.py | Improves UTF8Array handling in SimpleProxy; adds pandas string-column conversion + row-kernel operand plumbing; fixes kwarg forwarding for NumPy-returning kernels. |
| src/blosc2/ndarray.py | Adds StringDType→UTF8Array dispatch for constructors/asarray; fixes __radd__ order for string concatenation. |
| src/blosc2/lazyexpr.py | Improves miniexpr eligibility for string dtypes; fixes string-width inference via miniexpr probing; avoids SHUFFLE meta regression; routes UTF8Array expressions to span driver and rejects UTF8 operands in UDFs with actionable errors. |
| src/blosc2/indexing.py | Adds span coalescing and block-fraction gating to avoid bucket-index plans that are slower than scans. |
| src/blosc2/groupby.py | Updates comments/naming to UTF8Array terminology; keeps factorizer-based key logic. |
| src/blosc2/dsl_kernel.py | Adds string-syntax lowering and row-subscript rewriting so common pandas row-kernel idioms compile as DSL. |
| src/blosc2/dictionary_column.py | Adds decode caching (code→value), elementwise comparisons, and keeps identity hashing; avoids per-row dict-store reads. |
| src/blosc2/ctable_storage.py | Updates storage dispatch for UTF8Spec/UTF8Array and backend array creation. |
| src/blosc2/ctable_indexing.py | Adds utf8 rank indexing + persisted vocab; defaults index kind to FULL for rank-indexed flavours; records visibility epoch too. |
| src/blosc2/init.py | Exposes UTF8Array and conversion helpers (from_utf8, to_utf8, utf8_array). |
| plans/utf8-write-ingest-optim.md | Updates naming to UTF8Array; keeps ingest optimization plan current. |
| plans/utf8-string-support.md | Adds/records withdrawn compute-parity plan and rationale; documents what shipped instead. |
| plans/utf8-reads-filter-optim.md | Updates naming to UTF8Array; keeps read/filter optimization plan current. |
| plans/enhancing-ctable-phase3.md | Updates plan text naming to UTF8Spec/UTF8Array; records post-review fixes. |
| doc/reference/ctable.rst | Documents add_column(values=...), utf8 rank indexing, utf8 compute rule, and StringDType interop; adds API refs. |
| doc/reference/classes.rst | Adds UTF8Array to class reference listing. |
| doc/guides/pandas_engine.md | Documents row-subscript + control-flow compilation route and string-column support/limitations. |
| CMakeLists.txt | Bumps bundled/min required c-blosc2 to 3.3.0; updates miniexpr git tag. |
| bench/ndarray/jit-dsl-mandelbrot.py | Clarifies benchmark commentary around storage vs execution-tuning kwargs. |
| bench/chicago-taxi/README.md | Adds/expands string-ops benchmark documentation and performance notes. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+6751
to
+6755
| out: :ref:`NDArray` or :class:`UTF8Array` | ||
| A new :ref:`NDArray` made of :paramref:`array`, or the original | ||
| array when a copy is not required. | ||
| array when a copy is not required. When the target dtype is NumPy's | ||
| variable-length ``StringDType``, a :class:`UTF8Array` is returned | ||
| instead -- see the Notes. |
The section led with dictionary and described string() as "short codes of near-uniform length", which undersells the fastest of the four types: at max_length=32 it reads in 23 ms against utf8's 35 and filters in 104 ms against 165, on 500k rows. Recast it around the question that actually decides the choice -- is the length bounded, and is the bound small -- and give 32 as the threshold with the measurements behind it, including where the advantage decays (overtaken between 64 and 128, by which point fixed-width is using 60x the memory). 32 is also the width a bare `str` annotation already picks, so the default is now presented as the fast path rather than a compatibility leftover. Two things the old text did not say and a reader needs: the per-row cost is paid on every read rather than only on disk, which is why the width matters at all; and the compressed sizes of string() and utf8() are within ~30% on low-entropy text but ~7x apart on high-entropy values, so neither figure generalises. Also states what over-running max_length actually does -- raises on write, never truncates -- since the availability risk, not data loss, is the reason to prefer utf8 when the bound is a guess. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The previous commit claimed a value exceeding max_length always raises rather than truncating. That is true only of the validated write paths (constructor, append, extend). extend(validate=False), col[i] = value, Column.assign() and add_column(values=) bypass validation and fall through to NumPy's U semantics, truncating at max_length with no error. Say which paths do which, and keep the conclusion the honest way round: a guessed bound fails as rejected rows on one path and as silently shortened strings on another, so do not lean on the check. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
values= went straight to astype(spec.dtype), and coercing to a
fixed-width dtype truncates an over-long string to max_length rather
than complaining -- so add_column("c", string(max_length=8),
values=["abcdefghijklmnop"]) stored 'abcdefgh' with no error.
Route it through validate_column_values(), the same check extend() runs,
so it covers every declared constraint rather than just string lengths:
a values= of 999 into an int64(le=100) column now raises too.
Only this path changes. col[i] = value and Column.assign() keep NumPy's
U semantics; they are a released API, and the bypass they have is the
one the numeric constraints have as well (col[i] = 999 on int64(le=100)
stores 999), so tightening them is a wider decision than this one.
add_column(values=) is new in this branch and has no users to break.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three of the four findings were real: - _utf8_filled() built [fill] * shape[0], a list of shape[0] pointers to one interned string, before the packer saw any of it. to_utf8() takes any iterable, so itertools.repeat() streams it instead: peak memory for zeros(2_000_000, dtype=StringDType()) drops from 17.3 to 2.1 MiB. - empty/zeros/ones/full/asarray return a UTF8Array when the target dtype is NumPy's StringDType but were annotated `-> NDArray`, so the typing contradicted both the docstring and the behaviour. Widened to `NDArray | blosc2.UTF8Array`. The fourth -- that a column named after a DSL function or an index symbol could compile wrongly -- does not reproduce. A column named `sqrt` alongside a real np.sqrt() call in the same kernel gives the right answer, because operands and calls are distinguished by syntactic position; same for a column named `_i0`. Added a test pinning that rather than changing _param_for() to avoid a collision that does not occur. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
delete() tombstones rows in place and only decrements the live count, so live rows can sit past it. Both rank-index builders sized themselves by that count, which cost the dictionary index live rows in equality results and left a utf8 index permanently stale -- built, paid for, and never consulted. utf8 takes len(col), which carries no capacity padding and is the length the staleness check already compares against; dictionary takes the live-data watermark, since its own __len__ is the slot capacity. Also drop the null_rank == 0 branch in the utf8 != null-exclusion: it resolved to an always-empty lookup range, so != returned True for every row of an all-null column. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The bucket-selectivity gate had a separate branch for the case where a bucket spans a whole block, and it measured the wrong thing: the fraction of chunks holding any selected bucket, or a flat 1.0 for a 1-D mask. Both overstate the cost, so the gate declined the selective queries a bucket index exists to serve. The branch was also unnecessary -- with one bucket per block the grouping below it reduces to the mask itself and already gives the right answer, so it just goes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
raise type(e)(msg) assumes a one-argument constructor. The trace hint is attached to whatever the traced function raised, so any exception needing more arguments -- or refusing a bare string -- surfaced as a TypeError about that constructor, and the real failure was lost. add_note() keeps the exception's type, args and traceback intact. The guidance still prints with the traceback, but it is no longer part of str(e), so the kwargs test reads the notes as well as the message. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The inferred width follows the operand dtypes, but the cache was validated against the expression text alone. lazyexpr(expr, operands) rebinds in place and leaves that text untouched, so rebinding wider operands was answered from the narrower build: the output container came out too small and the concat truncated (or refused to evaluate under strict_miniexpr). The dtypes are collected before the check, which costs an attribute read per operand; what the key still protects is the miniexpr compile. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
86d466f made dotted utf8 leaves filterable and updated the string-type guide, but left this docstring claiming they are unsupported. Nothing replaces the sentence: it described a limitation that no longer exists, and the paragraph around it already lists what utf8 columns support. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The absolute 4 MiB peak bound tracked the platform's baseline allocation rather than the behaviour under test: Windows CI reported 6.2 MiB for the same streamed build that peaks at 2.1 MiB here, so the test failed there while nothing had materialized a fill list. Compare the peak at two sizes instead. The streamed build is flat in shape[0] -- 2.05 MiB from 250k rows to 4M here -- while a materialized list adds 8 bytes per row, ~27 MiB across the two sizes used. The baseline cancels, so the margin does not have to be tuned per platform. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 85 out of 87 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (3)
plans/utf8-write-ingest-optim.md:45
- This plan references a module path that no longer exists in the tree. The UTF8 implementation lives in
src/blosc2/_utf8_array.py, so the parenthetical here should be updated to avoid sending readers to a nonexistent file.
`UTF8Array` ingest (`src/blosc2/utf8_array.py`) is far slower than the
plans/utf8-reads-filter-optim.md:43
- This plan points to
src/blosc2/utf8_array.py, but the UTF8 implementation is now insrc/blosc2/_utf8_array.py. Updating the path will keep the plan actionable for future readers.
Every utf8 bulk read funnels through `UTF8Array._read_persisted_span`
(`src/blosc2/utf8_array.py`), which ends in a per-row Python loop:
plans/enhancing-ctable-phase3.md:152
- This implementation note still references
src/blosc2/utf8_array.py, but the code now lives insrc/blosc2/_utf8_array.py. Updating the filename here will prevent confusion when cross-referencing the plan against the current tree.
- What landed: `UTF8Spec`/`blosc2.utf8()` in `schema.py` (kind `"utf8"`,
registered in `schema_compiler._KIND_TO_SPEC`); new `src/blosc2/utf8_array.py`
with the `UTF8Array` adapter; storage dispatch in all four `TableStorage`
backends; sentinel-null wiring; guards for the not-yet-supported operations;
The summaries are built over the column's physical array, so a tombstoned row still contributes its value to its block's extrema. Keying the guard on the visibility epoch only caught rows deleted after the build; a delete followed by create_index() left the epoch matching and the shortcut on, and min() then reported a deleted row's value (0 instead of 1000 over arange(100_000) minus the first thousand rows, and symmetrically for max()). The same mismatch reached the straddling-block rescan, which indexes rows logically while the summary blocks it complements are physical. Both line up exactly while every slot below the watermark is live, so that is what the guard now asks. built_visibility_epoch had no other reader and goes with it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
utf8_span_eval took the first non-None sentinel it found and stamped it on every null row of the result, so an expression over two utf8 operands with different null_values relabelled one operand's nulls as the other's -- and which one won depended on the operand mapping's order. A result column has one sentinel, so there is no answer here to pick; refuse instead, and only where it shows. A boolean result forces nulls False whatever they were spelled as, and keeps working. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Main additions
Fixed-width
<Unand bytesSarrays now run string operations on miniexpr instead offalling back to NumPy: concatenation,
upper/lower, thestripfamily,removeprefix/removesuffix,replace,substr,split_part.@blosc2.dsl_kernelaccepts method syntax(
name.lower()) and tuple unpacking (before, after = desc.split(sep, 1)), which are loweredto the DSL grammar. Result width is inferred by miniexpr and the container allocated from it,
so nothing truncates —
.dtypemay be wider than NumPy's exact answer, never narrower.utf8()columns reach parity on the whole query surface:where()including stringfunctions,
sum(where=),sort_by,group_by, andcreate_index. Scalar predicates areanswered by a raw-byte scan over the offsets/blob with no row decoded. Indexing works by
deriving an alphabetical int32 rank per row — the trick the dictionary index already used —
which drives the existing numeric machinery unchanged:
sorted_slicetop-100 goes 458 → 43 msat 1M rows,
sort_byreaches parity with an indexed<Ucolumn, and the index is the cheapestof the three to build (277 ms against 867 ms). Persisting the sorted vocabulary makes a literal
→ rank lookup one
searchsorted, so==goes 29.0 → 5.5 ms and<34.6 → 5.5 ms.Compute is deliberately not at parity, and that is now a published rule rather than an
absence: utf8 stores and filters; fixed-width computes.
blosc2.from_utf8()/blosc2.to_utf8(),add_column(values=)andColumn.assign()on varlen columns make theround trip a two-liner, and every refusal names the column and prints the recipe. The
alternative — full compute parity — would have delivered an API indistinguishable from
<Urunning 3–5× slower, paid for with a
StringDType-in-schema round trip whose failure mode isa table that will not reopen.
plans/string-flavours-assessment.mdhas the measurements thedecision rests on.
Silent-wrong results fixed along the way
These were found while measuring and are independent of the feature work; most affect every
indexable dtype, not just strings:
utf8_arr == "hello"returned PythonFalse— object identity, because the containerdefined no comparison operators. Same on
DictionaryColumnand the varlen scalar columns.min()/max()read from a column index were wrong on two counts: capacity padding leakedinto the block summaries, and
delete()bumped a visibility epoch nothing recorded.kind=BUCKETcost more than the scan it replaced (float646.3 → 77.9 ms), becausescattered matches re-decompressed blocks and the planner gated on bucket selectivity while
the cost is paid in blocks.
sort_byat 1M rows took 235 s.dictcol != valueraisedIndexErroron any table with capacity padding.add_computed_column(name, kernel, inputs=["utf8_col"])was accepted and then broke thetable: every read and
str(table)raised afterwards.Worth a reviewer's eye
create_indexonutf8()/dictionary()now defaults to and requireskind=FULL; anexplicit other kind raises rather than building an index nothing can consult. Erroring
because there is no workload where those kinds help, and relaxing an error later is
non-breaking.
where("c == 'x'")string form still bypasses the index on both flavours,deliberately — substituting a precomputed mask measured slower (22.9 → 28.7 ms). The real
fix needs
plan_queryto consume index positions.blosc2.utf8(null_value="\x00")is now rejected: NumPy 2.4 does not match a lone NULagainst a
StringDTypearray, so that sentinel would silently stop marking nulls.Utf8Array/Utf8Specrenamed toUTF8Array/UTF8Spec, old names kept as aliases.blosc2_ext.pyx(a leak on the miniexpr udata allocation-failure paths),so CI has to rebuild the extension.