[mkitti]

Summary

Adds c-blosc2 as an opt-in build backend for the Blosc codec (-Dblosc_backend=blosc2). The default stays c-blosc 1.x, so nothing changes for existing builds.

Chunks written through this backend are in the Blosc1 format and decodable by C-Blosc 1.x. This is the point of the PR and it is not true of stock c-blosc2. Related: #864, #865, #854, #870, #463.

Forward compatibility

c-blosc2's README says its buffers are not readable by C-Blosc 1.x. I checked whether that still holds through the blosc1_* API (c-blosc 1.21.6 vs c-blosc2 3825863, 735 cases: blosclz/lz4/lz4hc/zlib/zstd × no/byte/bit shuffle × typesizes 1–16 × sizes 1 B–100 KB, including sizes that are not a multiple of the typesize):

writer → reader	decoded
stock c-blosc2 (blosc1_compress, with or without BLOSC_BLOSC1_COMPAT, or blosc2_compress_ctx) → C-Blosc 1.x	0
C-Blosc 1.x → stock c-blosc2	all except 56 bit-shuffled buffers with leftover bytes (a reader bug)
c-blosc2 with #815 → C-Blosc 1.x	all
C-Blosc 1.x → c-blosc2 with #815	all

The gap is in the writer: the format version byte (5 vs 2), the bitshuffle layout for blocks whose element count is not a multiple of 8, and the block-split threshold. The reader bug is fixed upstream (issue #812, PR #813, merged). The writer is merged upstream in Blosc/c-blosc2#815, which adds an opt-in blosc2_ctx_set_blosc1_format() (and makes BLOSC_BLOSC1_COMPAT enable it for blosc1_compress()); the output decodes with C-Blosc 1.x but is not byte-identical to what it writes. Only plain shuffle/bitshuffle with blosclz, lz4, lz4hc, zlib and zstd are supported in this mode; anything else is an error.

#815 is merged but not yet in a c-blosc2 release, so the c-blosc2 wrap points at the merge commit on c-blosc2 main (a98177412d3f). That has to move to an upstream release before numcodecs is released with this backend; packages/c-blosc2-blosc1-compat/TODO.md lists the steps. The package is self-contained (no numcodecs imports) and can be split out. See its README.

What changed

• packages/c-blosc2-blosc1-compat/: meson build for c-blosc2 (upstream is CMake only), a compatibility check (tests/run.py), and a tool to keep the wrap's copy of the build files in sync.
• subprojects/: a git wrap for c-blosc2 (main, pinned to a commit, with the meson build copied in through packagefiles) and for zstd, lz4 and zlib from WrapDB.
• blosc_backend option. zstd, lz4 and zlib are resolved separately from Blosc, so the zstd and lz4 codecs share them. -Dblosc_zstd=disabled / -Dblosc_zlib=disabled leave them out of Blosc only.
• src/numcodecs/_blosc_shim.h gives both libraries one C API, which blosc.pyx uses. numcodecs.blosc.LIBRARY reports which one is in use.
• blosc.pyx also picks up the frame validation and empty-buffer handling from Fix empty Blosc buffer round-trips #854, the zero-itemsize fix from Use a typesize of 1 in Blosc for zero-itemsize buffers #870, and clamps a requested blocksize to the buffer size and a multiple of the typesize as C-Blosc 1.x does (c-blosc2 writes unreadable chunks otherwise).
• Tests: Blosc1-format output, the shuffle round-trip from perf(blosc): use NEON SIMD for byte-shuffle on aarch64 #864, Fix empty Blosc buffer round-trips #854's frame tests, Use a typesize of 1 in Blosc for zero-itemsize buffers #870's zero-itemsize test, new tests for the decompress errors and blocksize handling, and tests that the wrap's copy of the build files and the UPSTREAM pin stay consistent. Compressors a build leaves out are skipped.
• .github/workflows/blosc2-compat.yaml: checks the wrap copy of the build files is in sync, runs the forward-compat check against a real C-Blosc 1.21.6, runs the whole test suite on the c-blosc2 backend (Linux x86_64 and arm64, macOS arm64, Windows, and free-threaded 3.14t).

Performance

#864's NEON copy is not needed for this backend: c-blosc2 builds its own NEON shuffle on aarch64 (the shuffle_neon symbols are in the extension). Single thread, Apple Silicon, 1 MiB arrays, SHUFFLE, decode, c-blosc 1.x → c-blosc2: lz4 uint16 1.5 → 4.6 GB/s, lz4 float32 1.6 → 8.9, blosclz uint16 1.8 → 20.4, blosclz float64 2.3 → 7.1. Encode is mixed (lz4 uint16 725 vs 1180 MB/s, others comparable or faster); one machine, not explained.

Notes for reviewers

• CI on the default backend is unchanged and passes; the new job covers the c-blosc2 backend.
• c-blosc2 has no snappy, and the wrapped zstd needs a C++ compiler (as in build: replace c-blosc git submodule with meson wrap subprojects #865).
• This does not remove the c-blosc submodule or change CI for the default backend; that belongs with build: replace c-blosc git submodule with meson wrap subprojects #865. Expect a rebase between this PR, perf(blosc): use NEON SIMD for byte-shuffle on aarch64 #864 and build: replace c-blosc git submodule with meson wrap subprojects #865: all touch the blosc part of src/numcodecs/meson.build and meson.options. perf(blosc): use NEON SIMD for byte-shuffle on aarch64 #864 is still relevant for the default c-blosc 1.x backend.
• pyproject.toml's license-files still lists the c-blosc submodule's licenses, so a c-blosc2 wheel does not yet ship c-blosc2's own. build: replace c-blosc git submodule with meson wrap subprojects #865 reworks this.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (54fc274) to head (492ab11).

Additional details and impacted files

@@            Coverage Diff            @@
##              main      #879   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           27        27           
  Lines          914       914           
=========================================
  Hits           914       914           

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[mkitti]

Performance: c-blosc 1.x vs c-blosc2 backend

Single thread (numcodecs.blosc.use_threads = False), compress(array, cname, 5, SHUFFLE), MB/s of uncompressed data, best of 40 runs after one warm-up. Apple Silicon (aarch64, macOS), Python 3.13, release build. Both backends built from this branch with default options (-Dblosc_backend=blosc1 is c-blosc 1.21.7.dev; blosc2 is c-blosc2 3825863 with the patch series). Ratio is uncompressed / compressed size.

cname	data	ratio 1.x → c-blosc2	encode 1.x → c-blosc2	decode 1.x → c-blosc2
lz4	uint16, 1 MiB	2.2 → 2.5	1180 → 725	1541 → 4641
lz4	float32, 1 MiB	1.4 → 1.4	1782 → 2981	1614 → 8883
lz4	float64, 1 MiB	1.2 → 1.2	1821 → 3721	2667 → 7406
lz4	uint16, 16 MiB	2.2 → 2.5	1073 → 672	1544 → 4352
blosclz	uint16, 1 MiB	2.0 → 2.0	1964 → 1158	1849 → 20410
blosclz	float32, 1 MiB	1.4 → 1.3	1154 → 1248	1305 → 7840
blosclz	float64, 1 MiB	1.2 → 1.2	1121 → 1691	2340 → 7059
blosclz	uint16, 16 MiB	2.0 → 2.0	1930 → 1120	1868 → 20243

What this shows:

• Decode is 3x to 11x faster on c-blosc2. c-blosc 1.x has no NEON shuffle, so its dispatcher falls back to scalar byte shuffle, which is the problem perf(blosc): use NEON SIMD for byte-shuffle on aarch64 #864 describes (about 2.5 GB/s here). c-blosc2 ships its own NEON shuffle (shuffle_neon / unshuffle_neon are in the built extension), so no extra NEON code is needed for this backend.
• Encode is mixed: faster for float32/float64 with lz4 and for blosclz float64, slower for uint16 (lz4 725 vs 1180 MB/s, blosclz 1158 vs 1964). I have not found why. A guess I have not tested is the block-split rule that the Blosc1 output mode forces; lz4 uint16 also compresses better (2.5 vs 2.2), which fits a different split.
• This is one machine, one run of the script, and my own synthetic data (cumulative sums, so that shuffle matters), so it is not comparable to the table in perf(blosc): use NEON SIMD for byte-shuffle on aarch64 #864, which used different data. x86_64 (SSE2/AVX2) has not been measured.

Script

import sys, time
import numpy as np
sys.path.insert(0, sys.argv[1])
import numcodecs.blosc as b
b.use_threads = False
print(b.LIBRARY, b.VERSION_STRING)
def mbs(f, n, reps=40):
    f(); best = 1e9
    for _ in range(reps):
        t = time.perf_counter(); f(); best = min(best, time.perf_counter() - t)
    return n / best / 1e6
rng = np.random.default_rng(0)
cases = [("uint16 1MiB", np.cumsum(rng.integers(0, 3, 2**19)).astype('u2')),
         ("float32 1MiB", np.cumsum(rng.normal(scale=1e-3, size=2**18)).astype('f4')),
         ("float64 1MiB", np.cumsum(rng.normal(scale=1e-3, size=2**17))),
         ("uint16 16MiB", np.cumsum(rng.integers(0, 3, 2**23)).astype('u2'))]
for cname in ("lz4", "blosclz"):
    for name, a in cases:
        raw = a; n = a.nbytes
        enc = b.compress(raw, cname.encode(), 5, b.SHUFFLE)
        e = mbs(lambda: b.compress(raw, cname.encode(), 5, b.SHUFFLE), n)
        d = mbs(lambda: b.decompress(enc), n)
        _, _, memcpyed = b._cbuffer_metainfo(enc)
        print(f"{cname:8s} {name:13s} ratio {n/len(enc):5.1f} memcpy={memcpyed} encode {e:7.0f}  decode {d:7.0f} MB/s")

Run it as python bench.py <directory containing the built numcodecs> for each backend.

Two traps I hit while writing it, in case someone repeats the measurement: passing bytes gives typesize 1, so the shuffle does nothing, and data that does not compress is stored raw (check blosc._cbuffer_metainfo(enc)[2], the memcpy flag), which makes decode look like 30+ GB/s.

[mkitti]

Follow-up to the performance comment above: the encode slowdown it shows for 2-byte data, and where it comes from.

What is slower. On the c-blosc2 backend, encoding 2-byte types (uint16) is about 25% to 45% slower than with c-blosc 1.x at the same codec, level and shuffle (lz4 725 vs 1137 MB/s, blosclz 1158 vs 1964 in the table above). 4- and 8-byte types are comparable or faster, and decode is much faster everywhere I measured. Against c-blosc 1.x with #864's NEON applied, c-blosc2 is slower on encode in most rows (see my comment on #864).

Cause: a different automatic block size, not slower code. In a C-level test (1 MiB, typesize 2, clevel 5, one thread), c-blosc 1.x stores 256 KiB blocks and c-blosc2 stores 128 KiB blocks:

	block size	lz4 ratio	lz4 encode	blosclz encode
c-blosc 1.x	256 KiB	2.19	~925 MB/s	2024 MB/s
c-blosc2 default	128 KiB	2.52	~735 MB/s	1169 MB/s
c-blosc2 forced to 256 KiB	256 KiB	2.19	1777 MB/s	2110 MB/s

At the same block size c-blosc2 produces the same ratio and is as fast or faster. The two libraries compute the default differently for splittable codecs: c-blosc 1.x uses min(128 KiB, 256 KiB) * typesize at clevel 5, and c-blosc2 (blosc_stune_next_blocksize in blosc/stune.c) uses a fixed 64 KiB for clevels 4 to 6, times the typesize. The smaller block gives a better ratio and, for typesize 2, a slower encode. It looks deliberate on their side, so I filed it as a question: Blosc/c-blosc2#814 (with a reproduction using python-blosc and python-blosc2).

Correction: I first guessed on #864 that the Blosc1 output mode's block-split rule was responsible. That was wrong. Stock c-blosc2 with its default Blosc2 output is equally slow, and the patched Blosc1 mode is within a few percent of it.

What this means for the PR.

• It does not affect correctness or the default backend; the opt-in c-blosc2 backend just makes a different speed/ratio choice for small typesizes at the default block size.
• Blosc(blocksize=...) already lets a user pick the block size explicitly; with blocksize=262144 on 2-byte data the c-blosc2 backend matches c-blosc 1.x's ratio and is faster.
• An option I have not implemented: for blocksize=0 the c-blosc2 backend could choose c-blosc 1.x's block size itself, so output matches c-blosc 1.x chunks more closely. That would give up c-blosc2's better ratio for small types, and it would need checking across typesizes, levels and buffer sizes, so I would rather decide it after upstream answers Documentation cleanup: fix typos, grammar, and naming consistency #814.

Caveats: one machine (Apple Silicon), one buffer size for the block-size experiment, synthetic data, no x86_64.

Drafted with Claude Code.

[d-v-b]

so did they avoid supporting the old blosc1 format header to encourage people to switch to blosc2?

[mkitti]

It did not seem to encourage us very much here, did it?

[d-v-b]

this is a big change so I'd like some eyes from @zarr-developers/python-core-devs.

my pov: this effort already revealed one bug in blosc2. I wonder if there are more? for me, a perf improvement isn't worth a moderate chance of data corruption / bugs. I personally am not qualified to evaluate the c code or the build backend changes so I would like to see some very comprehensive property-based tests that fuzz the new codec and ensure that the range of bytes payloads, with important edge cases, that we see in zarr come out byte-identical after this swap.

[martindurant]

Thanks for the ping to maintainers. I am surprised that there would be a push to implement this in C with a massive PR rather than call the python or C API - is the overhead really that crushing? This has always puzzled me about numcodecs specifically for blosc; at least the cython-on-Clib codecs are reasonable.
I don't know about the bug mentioned above or potential others, but I think for simplicity and ease of packaging/distribution, it would be better to call public APIs of installable packages.

Generally, the comparison with what blosc does internally versus zarr's sharding filter is super interesting for me. (yes, I realise here we are in numcodecs/zarr1 land) I feel like the two libraries are sort of doing the same thing and it may be better to (hope to) converge. blosc has done some interesting things with in-memory compressed arrays and operation fusion that zarr has not been interested in or left to other frameworks.

cc @FrancescAlted , might find this interesting.

[mkitti]

@martindurant The C code patches included in this pull request add a function which c-blosc2 nor its Python wrapping actually does by itself, which is writing write conpatible c-blosc (v1) frames that c-blosc (v1) can decode.

Forward compatibility has long been a blocker here inhibiting the migration to c-blosc2.

Other pull requests started to pull code from c-blosc2 for ARM Neon support, as an example, which I think is more unsustainable than just using c-blosc2 which already maintains such support.

Regardless of depedency and integration mechanism, continued dependence on c-blosc (v1) has become unsustainable, untenable, and irresponsible in any form.

[d-v-b]

the blosc v1 codec is baked in to the zarr v3 spec. so we are on the hook for supporting it.

[mkitti]

this is a big change

As currently written this pull request actually changes nothing by default. The default is to use the c-blosc (v1) "backend". It must be enabled by a compiler flag whichis disabled by default.

This pull request allows for the very testing you are requesting to begin.

[d-v-b]

This pull request allows for the very testing you are requesting to begin.

I think what I was saying is that, given the scale of the intended change (swapping out our vendored blosc1 for a patched blosc2), I would need to see those tests before I can think about supporting this effort.

[mkitti]

the blosc v1 codec is baked in to the zarr v3 spec. so we are on the hook for supporting it.

The blosc v1 codec is distinct from the c-blosc (v1) code base, which is seeing less and less developer attention. I have submitted issues and pull requests to the v1 code base in the past but decided it was no longer worthwhile to do about a year ago.

The patches included here show a path to supporting the v1 codec from the c-blosc2 code base. That said the lack of upstream support for writing the c-blosc v1 format suggest that we should begin deprecating writing with that codec.

[d-v-b]

I agree with @martindurant that the best outcome would be for us to just import all this stuff from a published package. @FrancescAlted would applying the patches here to blosc2 be in scope for the project? then we could just import blosc2 from pypi, which would be far simpler than what we have here.

[FrancescAlted]

so did they avoid supporting the old blosc1 format header to encourage people to switch to blosc2?

Just let me make clear that we are not supporting old blosc1 very actively mainly for two reasons:

1. All action is happening in blosc2, so there is not much point for us in keep maintaining blosc1 for performance improvements (fixing important bugs is a different thing, and we tried to keep doing this with our limited resources).
2. Contrarily to blosc2, blosc1 has not received any funding for more than 10 years now, strongly affecting our maintenance work.

Having said that, it is completely true that we encourage people to switch to blosc2. We really think that this is the way to go forward, and for me, adding support for blosc2 in zarr makes a lot of sense. In blosc2 we are actively supporting backward compatibility with blosc1, and I think this could facilitate a lot the transition to blosc2.

[FrancescAlted]

I agree with @martindurant that the best outcome would be for us to just import all this stuff from a published package. @FrancescAlted would applying the patches here to blosc2 be in scope for the project? then we could just import blosc2 from pypi, which would be far simpler than what we have here.

Absolutely. But IMO, providing a way to generate blosc1-compatible format inside blosc2 has little sense, because for me, focusing on the new format should bring more value for users (it is way more flexible) than keep trying using just the old one. Having said that, if you find that this is paramount for you, I am fine with that (within our finite resources).

[mkitti]

In blosc2 we are actively supporting backward compatibility with blosc1, and I think this could facilitate a lot the transition to blosc2.

Thank you for replying, @FrancescAlted . What is holding adoption of c-blosc2 back here is that c-blosc2 does not support writing c-blosc v1 compatible chunks. What I have done here is provide patches for c-blosc2 which allow for that - write chunks that c-blosc1 can read to enable the transition to the c-blosc2 code base. This seems more practical than trying to backport ARM Neon patches from c-blosc2 to c-blosc v1.

The question Davis has is "Would you accept patches that enable writing c-blosc v1 formatted chunks in c-blosc2?"

[FrancescAlted]

In blosc2 we are actively supporting backward compatibility with blosc1, and I think this could facilitate a lot the transition to blosc2.

Thank you for replying, @FrancescAlted . What is holding adoption of c-blosc2 back here is that c-blosc2 does not support writing c-blosc v1 compatible chunks. What I have done here is provide patches for c-blosc2 which allow for that - write chunks that c-blosc1 can read to enable the transition to the c-blosc2 code base. This seems more practical than trying to backport ARM Neon patches from c-blosc2 to c-blosc v1.

The question Davis has is "Would you accept patches that enable writing c-blosc v1 formatted chunks in c-blosc2?"

Yep. Although this does not make a lot of sense to me. See my reply to him above.

[mkitti]

I have opened a pull request to apply the last three patches to c-blosc2 directly.

Blosc/c-blosc2#815

[mkitti]

Re: the request for tests that the payloads zarr sees come out "byte-identical after this swap" (comment). I added property-based tests and CI for compatibility with C-Blosc 1.x (e014c8f), but they check that the decoded bytes are identical, not that the compressed bytes are. The compressed bytes will not be identical, and cannot be made so without reimplementing C-Blosc 1.x's choices inside c-blosc2.

Why. The Blosc1 format specifies how a chunk is decoded, not how it must be encoded. A writer is free to choose the block size, whether to split blocks, and what the codec emits. Any C-Blosc 1.x reader decodes every valid choice. c-blosc2 makes different choices from c-blosc 1.x in the places below, and the Blosc1 mode in Blosc/c-blosc2#815 deliberately only changes what a C-Blosc 1.x reader needs (the version byte, the bitshuffle layout of blocks whose element count is not a multiple of 8, and never splitting a block the reader would not split).

• Default block size. For 1 MiB of uint16 with lz4 at level 5, c-blosc 1.x writes 256 KiB blocks and c-blosc2 writes 128 KiB blocks (Slower default encoding than C-Blosc 1.x for 2-byte types: smaller automatic block size Blosc/c-blosc2#814). That one alone changes the block count, the offsets table and every compressed stream after the first block.
• Block splitting. c-blosc2 decides when to split a block into one stream per byte of the typesize with its own heuristic (codec, level, whether a shuffle is used, element count). c-blosc 1.x uses a different rule. The split flag is in the header, so the bytes differ even when the data decodes the same.
• The compression step itself. The compressed bytes also depend on the codec library and the heuristics around it (acceleration, when a block is stored uncompressed, the versions of the bundled blosclz, lz4 and zstd). I have not broken that part down further or established which of these accounts for the 392 cases.

How often, in practice. I compared the two writers on the 1050 deterministic cases of the grid test with at least 32 bytes (blosclz, lz4, lz4hc, zlib, zstd; 3 shuffles; typesizes 1 to 300; sizes 32 B to 100 KB; level 5, automatic block size). 213 are byte-identical and 837 differ:

difference	cases
same block size and split flag, different compressed size	392
different split flag	335
different block size	105
same size, different bytes	5

For the 1 MiB uint16 example above, numcodecs on c-blosc2 gives 416,503 bytes and C-Blosc 1.x gives 481,483.

This is also not specific to c-blosc2: the compressed bytes of any Blosc writer depend on its codec versions and heuristics, so they are not something readers can rely on, and a version bump of c-blosc 1.x's own codecs can change them too.

What the tests do guarantee, on every backend (c-blosc 1.x as a control, c-blosc2 on Linux x86_64/arm64, macOS and Windows), with python-blosc (which bundles C-Blosc 1.x) as the reference, in both directions:

• what numcodecs writes decodes with C-Blosc 1.x to the original bytes, and what C-Blosc 1.x writes decodes with numcodecs;
• the chunk is format version 2 and its header has the right nbytes, cbytes and typesize, with a typesize above 255 stored as 1 as C-Blosc does;
• this holds for every compressor, level 0 to 9, all three shuffles, typesizes 1 to 300, buffer sizes at the edges that matter (0, 32, multiples of 8 and 128 elements, bytes left over after the last element), explicit and automatic block sizes, and zeros, constant, random, ramp, periodic, sparse and sorted data;
• both numcodecs code paths (blosc's global context, and numcodecs' own context used with use_threads=False, in other threads and in forks) are covered. A first version missed the second path; I found that by making that path write stock c-blosc2 chunks, which made 1506 of the tests fail.

It runs 300 examples per property on a pull request and 5000 weekly, and it can be switched off without a code change (repository variable BLOSC1_COMPAT_CI=false). If a byte-for-byte comparison with c-blosc 1.x output would still be useful to you, tell me which output you want to pin (for example a fixed block size and split mode) and I can add that as a separate, narrower test; it would have to fix those choices explicitly.

Drafted with Claude Code.

[d-v-b]

thanks for adding those tests. the inability to match byte-for-byte is probably not a blocker, given the advantages of getting on a more secure dependency footing, but it is a change that will affect people and potentially require them to update their code.

Anyone who assumes a fixed relationship between the blosc configuration and a compressed payload will need to update that assumption after that change, and potentially alter any code paths that rely on the size or contents of the compressed payload. So we probably need to socialize this change a bit in advance, so folks can prepare for it. I know a few tests in zarr that I suspect will break because of this. I'm fine dealing with that but it's a cost to be aware of.

[mkitti]

Follow-up to my comment on byte-identical output: what the public API can do about it, and one result worth recording, which is what the split mode does.

Finding: NEVER gives the most byte-identical chunks of the four split modes. On the same 1050-case grid as before (blosclz/lz4/lz4hc/zlib/zstd x 3 shuffles x typesizes 1 to 300 x 32 B to 100 KB, level 5), with the split mode set through BLOSC_SPLITMODE for the whole process, the number of chunks that are byte-for-byte what C-Blosc 1.x writes (python-blosc, C-Blosc 1.21.7.dev) is:

BLOSC_SPLITMODE	identical (of 1050)
unset / FORWARD_COMPAT (the default)	213 (20%)
ALWAYS	302
AUTO	373
NEVER	396 (38%)

Passing C-Blosc 1.x's own block size to c-blosc2 as blocksize changed nothing on this grid (only ALWAYS moved, 302 to 321), because the buffers here are small enough that the automatic block sizes mostly agree. Picking the best of the four modes for each case reaches 443 (42%), which is the ceiling for this knob on this grid.

Why a single mode is not the answer. C-Blosc 1.x decides per codec: it splits every codec except zstd (provided the typesize is at most 16 and the block has at least 128 elements, which the Blosc1 mode in Blosc/c-blosc2#815 already enforces on top of whatever mode is requested). I checked the flag in real chunks: blosclz, lz4, lz4hc and zlib split, zstd does not. c-blosc2's default (FORWARD_COMPAT) uses its own heuristic that depends on the level and on whether a shuffle is used, which is why it is the worst of the four here. NEVER is the right choice for zstd and the wrong one for the other four codecs. I have not decomposed why it comes out ahead of ALWAYS in aggregate on this grid; cases where splitting is impossible are identical under every mode, so they are not the reason. Applying C-Blosc 1.x's rule (ALWAYS for blosclz/lz4/lz4hc/zlib, NEVER for zstd) and its block size formula (compute_blocksize in blosc.c, reimplemented in the test script) on a broader grid of 4320 cases (adds 1 MiB buffers, levels 1, 5 and 9, and random as well as compressible data):

	identical (of 4320)
numcodecs as it is (default split mode, automatic block size)	581 (13.4%)
split mode chosen per codec	1296 (30.0%)
... and C-Blosc 1.x's block size	1456 (33.7%)

For the 1 MiB buffers alone it goes from 87 to 291 of 540. The reimplemented block size formula agrees with C-Blosc 1.x's header in 4212 of the 4320 cases.

What does not move. Two things the API cannot change keep it from going further:

• blosclz writes a different stream. Even with the same block size and split decision the compressed bytes differ (1134 vs 1112 bytes in one case I opened). Of the blosclz cases in the first grid, at most 34 of 210 were identical under any split mode.
• Incompressible data. C-Blosc 1.x sets the "memcpy" flag for the whole chunk when compression gains nothing; c-blosc2 stores the block uncompressed inside a normal block. I tried emulating C-Blosc 1.x by re-encoding such chunks at level 0 and it made the identical count worse (1456 to 922), so I would not do that.

What I would and would not do with this. The tests in this PR check decoded bytes, which is what compatibility needs, and I do not think numcodecs should chase byte-identical output: the stream differences above are outside our control, and the remaining 66% differ for reasons the format allows. If it is still wanted, the cheapest step is to pass C-Blosc 1.x's split rule through the backend: numcodecs would call blosc1_set_splitmode() under the existing lock on the global path and set cparams.splitmode on the context path, by codec. I have not made that change. Note that BLOSC_SPLITMODE, like BLOSC_DELTA, is sticky: it sets a global that stays set after the variable is removed, so toggling it between calls in one process (as I did at first, and got confusing numbers) needs it set explicitly every time.

Caveats: one machine (x86_64), synthetic data, level 5 for the first grid and levels 1/5/9 for the second, and python-blosc's bundled C-Blosc 1.21.7.dev as the reference.

Drafted with Claude Code.

[martindurant]

Anyone who assumes a fixed relationship between the blosc configuration and a compressed payload

That doesn't seem like something we would promise? I don't know, but I would not be surprised if an update in Zstd, lz4 etc might change the default output for a given array.

[d-v-b]

Anyone who assumes a fixed relationship between the blosc configuration and a compressed payload

That doesn't seem like something we would promise? I don't know, but I would not be surprised if an update in Zstd, lz4 etc might change the default output for a given array.

No I don't think it's something we should promise. But it is an expected consequence of the change here, so worth keeping it in mind.

[mkitti]

All patches have been upstreamed.