Skip to content

Use NEON on Linux arm64 - #339

Open
MayCXC wants to merge 2 commits into
asg017:mainfrom
MayCXC:neon-linux-arm64
Open

MayCXC wants to merge 2 commits into
asg017:mainfrom
MayCXC:neon-linux-arm64

Conversation

@MayCXC

@MayCXC MayCXC commented Oct 23, 2026 •

Copy link
Copy Markdown

This builds on #322 and carries its commit: GCC cannot build the NEON code on main (#299), because l1_int8_neon() relies on implicit signed/unsigned vector conversions, and #322 makes them explicit. Once #322 is merged, this is the one commit after it.

The Makefile enables NEON only on macOS arm64, so a Linux arm64 build, the linux-aarch64 release binary among them, runs every distance's scalar loop. NEON is part of the arm64 baseline: GCC and Clang define __ARM_NEON for aarch64 with no -m flag, so the Linux block needs only the define, beside its AVX check and inside its guard against Android cross-compiles. OMIT_SIMD=1 still builds without it.

The define follows the compiler's target, $(CC) -dumpmachine, not uname -m: 32-bit Raspberry Pi OS runs a 64-bit kernel, so uname says aarch64 while gcc targets 32-bit ARM, which lacks the AArch64-only intrinsics the NEON kernels use. Under an aarch64 kernel, make loadable CC=arm-linux-gnueabihf-gcc fails when uname decides (implicit declaration of function 'vaddvq_f32', unknown type name 'float64x2_t') and builds a 32-bit ARM extension when the compiler does. For gcc and clang targeting aarch64 the build commands are the same either way.

Built with GCC 14 at -O3 on Linux arm64 (medians of 20 queries on a 100,000-row float[768] table, each build in its own process: the extension exports its internal functions, so a second build loaded into the same process runs the first one's KNN code):

metric k scalar NEON
L2 10 59.2 ms 19.6 ms
L2 100 65.8 ms 24.2 ms
cosine 10 85.3 ms 17.5 ms
cosine 100 90.6 ms 23.3 ms

What changes in the results, on 2,000 random pairs at each of six lengths from 8 to 1,536:

  • L1 and Hamming distances are bit-identical. The float32 L1 kernel adds in another order than the scalar loop, both in double, so the two can differ in the last digit on other inputs.
  • float32 L2 and cosine distances differ in the last digits, since the NEON kernels split their sums across 16 partial sums where the scalar loops keep one; for L2 the NEON result is closer to the exact value, a mean relative error of 3.0e-8 instead of 1.7e-7 at 768 dimensions. So a KNN query can order rows with nearly equal distances differently, as the macOS arm64 build already does; 4 of the 2,000 ranks of the k = 100 L2 queries above swapped such rows.
  • int8 cosine distances can differ in the last digit, since cosine_int8_neon() finishes in float and the scalar loop in double. int8 L2 distances differ only past about 1,536 random dimensions, where the scalar loop's float sum stops being exact; the kernel's 32-bit sum overflows in vec_distance_l2() past 33,026 elements at opposite extremes, as it already does on macOS arm64, and vec0 tables allow at most 8,192 dimensions.

The Test workflow's build-linux-aarch64-extension job already runs the suite on an arm64 runner, so it now tests the NEON build. Locally the suite passes on it (478 passed, 132 skipped, 204 snapshots).

Origin. This is photostructure/sqlite-vec@5fb0b3d (photostructure#6), with mceachen's photostructure/sqlite-vec@350c67a, which took the define from the compiler's target there and noted the L1 summation order. vlasky#21 turns NEON on for vlasky's fork. No open or closed PR or issue here turns NEON on for Linux arm64.

It merges cleanly with #338 and with #327 through #331, #336 and #337. It conflicts with stumpylog's #307 in the Makefile's Linux block, which #307 limits to x86_64 for its AVX change.

The Makefile enabled NEON only on macOS arm64, so a Linux arm64 build,
the linux-aarch64 release binary among them, ran every distance's
scalar loop. NEON is part of the arm64 baseline: GCC and Clang define
__ARM_NEON for aarch64 with no -m flag, so the Linux block needs only
the define, beside its AVX check and inside its guard against Android
cross-compiles. OMIT_SIMD=1 still builds without it.

The define follows the compiler's target, `$(CC) -dumpmachine`, not
uname, as mceachen's photostructure/sqlite-vec@350c67a has it: 32-bit
Raspberry Pi OS runs a 64-bit kernel, so uname says aarch64 while gcc
targets 32-bit ARM, which lacks the AArch64-only intrinsics the NEON
kernels use. Under an aarch64 kernel, `make loadable
CC=arm-linux-gnueabihf-gcc` fails when uname decides (implicit
declaration of vaddvq_f32, unknown type name float64x2_t) and builds a
32-bit ARM extension when the compiler does; for gcc and clang
targeting aarch64 the build commands are the same either way.

GCC cannot build the NEON code on main, because l1_int8_neon() relies
on implicit signed/unsigned vector conversions; this builds on asg017#322,
which makes them explicit.

Built with GCC 14 at -O3 on Linux arm64, a KNN query on a 100,000-row
float[768] table takes 19.6 ms instead of 59.2 ms for k = 10 and
24.2 ms instead of 65.8 ms for k = 100 with the L2 metric, and 17.5 ms
instead of 85.3 ms and 23.3 ms instead of 90.6 ms with cosine (medians
of 20 queries, each build in its own process: the extension exports
its internal functions, so a second build loaded into the same process
runs the first one's KNN code). L1 and Hamming distances were
bit-identical to the scalar loops' on 2,000 random pairs at each of six
lengths from 8 to 1,536, though the float32 L1 kernel adds in another
order than the scalar loop, both in double, so the two can differ in
the last digit on other inputs. Float32 L2 and cosine distances differ
in the last digits, since the NEON kernels split their sums across 16
partial sums where the scalar loops keep one; for L2 the NEON result is
closer to the exact value, a mean relative error of 3.0e-8 instead of
1.7e-7 at 768 dimensions. So a KNN query can order rows with nearly
equal distances differently, as the macOS arm64 build already does; 4
of the 2,000 ranks of the k = 100 L2 queries swapped such rows. Int8
cosine distances can differ in the last digit as well, since
cosine_int8_neon() finishes in float and the scalar loop in double.

The Test workflow's build-linux-aarch64-extension job already runs the
suite on an arm64 runner, so it now tests the NEON build. Locally the
suite passes on it (478 passed, 132 skipped, 204 snapshots).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants