[d-v-b]

🤖 AI text below 🤖

Summary

c-blosc 1.x only ships SSE2/AVX2 shuffle kernels. On aarch64 (Apple Silicon, AWS Graviton, …) its dispatcher always falls back to blosc_internal_(un)shuffle_generic, a scalar byte-by-byte loop. That loop dominates Blosc runtime on ARM: with shuffle disabled, the same build decodes lz4 at ~17 GB/s, but with shuffle enabled it decodes at ~2.5 GB/s.

This PR adds src/numcodecs/_blosc_neon/shuffle-generic-neon.c. On aarch64, meson compiles it instead of c-blosc's shuffle-generic.c. It defines the same two symbols:

• NEON kernels for type sizes 2, 4, 8 and 16, copied verbatim from c-blosc2's blosc/shuffle-neon.c (BSD-3, attributed in the header).
• c-blosc1's scalar code for other type sizes and for the tail of each block.

The c-blosc submodule is not modified, and c-blosc's dispatcher (shuffle.c) is unchanged. A new neon meson option (default auto) controls this. Bitshuffle is unchanged, since c-blosc2 also found its NEON bitshuffle no faster than scalar.

Compatibility

The output is byte-identical to the scalar build. The agent compared SHA-256 hashes of the compressed output from both builds over 1,452 combinations: type sizes 1–32 including odd sizes, 11 buffer lengths, 3 block sizes, shuffle and bitshuffle, lz4 and blosclz. The hashes matched and every case round-tripped. The existing backwards-compatibility fixtures also decode correctly.

Benchmarks

Apple M5, Python 3.14, single thread (use_threads = False), Blosc(cname, clevel=5, shuffle=SHUFFLE), MB/s of uncompressed data. The numbers were stable to within ~1% across two runs.

data	cname	encode before → after	decode before → after
uint16, 1 MiB	lz4	2665 → 5758	2557 → 16024
uint16, 16 MiB	lz4	2309 → 4274	2344 → 10497
float32, 1 MiB	lz4	2452 → 5021	2498 → 13198
float64, 1 MiB	lz4	2812 → 5793	4663 → 14264
uint16, 1 MiB	blosclz	1685 → 2462	2210 → 8338
float32, 1 MiB	zstd	257 → 273	2033 → 5038

Testing

• Adds test_shuffle_roundtrip_typesizes. It round-trips across type sizes, lengths that aren't multiples of the vector width, and block sizes, to cover the scalar tail.
• The full suite passes locally on macOS arm64.
• Not tested: Linux aarch64 and Windows ARM64. For MSVC, the source guard and the meson check accept _M_ARM64 as well as __ARM_NEON.

Notes for reviewers

• A longer-term alternative is to upstream these kernels to c-blosc 1.x. The file-substitution approach here avoids carrying a fork of the submodule in the meantime.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (54fc274) to head (1ff46e7).

Additional details and impacted files

@@            Coverage Diff            @@
##              main      #864   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           27        27           
  Lines          914       914           
=========================================
  Hits           914       914           

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[mkitti]

I really think we should see if c-blosc2 already does everything we need.

It can now read and write blosc1 data:
https://github.com/Blosc/c-blosc2/blob/38258631914de7f6ca2cf55c9675d771796ab96c/tests/test_blosc1_compat.c#L4

and they already have neon code:
https://github.com/Blosc/c-blosc2/blob/38258631914de7f6ca2cf55c9675d771796ab96c/blosc/bitshuffle-neon.c#L285

[d-v-b]

i was under the impression that blosc2 would not write data that a blosc1 reader could decode. if that's wrong, then blosc2 looks like a great idea

[mkitti]

I measured #864 against the c-blosc2 backend I propose in #879, since c-blosc2 builds its own NEON shuffle on aarch64. The short version: c-blosc2 is not uniformly faster than c-blosc 1.x with this patch. #864 recovers most of the NEON speedup for the default backend, and in several cases it is faster than c-blosc2, especially on encode.

Single thread (use_threads = False), compress(array, cname, 5, SHUFFLE), MB/s of uncompressed data, best of 40 runs after a warm-up, Apple Silicon, Python 3.13. Three builds from source, run back to back: 1.x is the current main with c-blosc 1.21.7.dev; 1.x + #864 is this PR at 1ff46e7 with the same c-blosc (04c06fb); c-blosc2 is c-blosc2 3825863 through the backend in #879. Two full runs agreed within about 5%; the table is the second.

cname	data	encode: 1.x → 1.x + #864 → c-blosc2	decode: 1.x → 1.x + #864 → c-blosc2
lz4	uint16, 1 MiB	1137 → 1690 → 705	1590 → 5829 → 4657
lz4	float32, 1 MiB	1729 → 3610 → 3047	1570 → 10050 → 8414
lz4	float64, 1 MiB	1839 → 3526 → 3651	2679 → 6282 → 6640
lz4	uint16, 16 MiB	1102 → 1574 → 700	1573 → 5882 → 4357
blosclz	uint16, 1 MiB	2044 → 4782 → 1161	1892 → 15280 → 20730
blosclz	float32, 1 MiB	1176 → 1773 → 1259	1358 → 4861 → 8126
blosclz	float64, 1 MiB	1132 → 1636 → 1686	2376 → 4891 → 8171
blosclz	uint16, 16 MiB	1968 → 4299 → 1141	1876 → 14440 → 20209

(Bold is the fastest of the three.)

• perf(blosc): use NEON SIMD for byte-shuffle on aarch64 #864 helps a lot: decode is 2.5x to 8x faster than stock c-blosc 1.x on this data, and encode is faster too, so it is worth merging for the default backend.
• c-blosc2 wins on blosclz decode (1.3x to 1.7x over 1.x + perf(blosc): use NEON SIMD for byte-shuffle on aarch64 #864) and is about equal for lz4 float64; it loses on lz4 decode for uint16 and float32 (4657 vs 5829, 8414 vs 10050).
• c-blosc2 is slower on encode in most rows, up to about 4x for blosclz uint16 (1161 vs 4782). Correction: my first version of this comment guessed this was the block-split rule used by the Blosc1 output mode in Optional c-blosc2 backend for Blosc that writes Blosc1-format chunks #879. That was wrong: stock c-blosc2 with its default Blosc2 output is just as slow, and the patched Blosc1 mode matches it within a few percent. In a C-level test (1 MiB, typesize 2, clevel 5, one thread) the cause is that c-blosc2 chooses a smaller automatic block size than c-blosc 1.x (128 KiB vs 256 KiB). At the same 256 KiB block size c-blosc2 gives the same ratio (2.19 for lz4) and encodes faster (lz4 1777 vs about 925 MB/s; blosclz 2110 vs 2024). The smaller default block compresses better (lz4 2.52 vs 2.19) at lower speed. So it is a different default block size, not slower code.
• So I would not describe the c-blosc2 backend as a performance reason to drop perf(blosc): use NEON SIMD for byte-shuffle on aarch64 #864. perf(blosc): use NEON SIMD for byte-shuffle on aarch64 #864 still makes sense for the default c-blosc 1.x backend whatever happens to Optional c-blosc2 backend for Blosc that writes Blosc1-format chunks #879.

Caveats: one machine, my own synthetic data (cumulative sums, chosen so that shuffle matters, with memcpy=0 checked), so this is not comparable to the table in the description of this PR. x86_64 was not measured. The benchmark script is in #879 (comment).

Drafted with Claude Code.