Repository navigation
Prebuilt: run the Linux CPU bundles on emulated CPUs (Intel SDE, qemu-aarch64) before shipping - #49
Open
danielhanchen wants to merge 41 commits into
Open
danielhanchen wants to merge 41 commits into
danielhanchen wants to merge 41 commits into
Conversation
…t on Linux The sysfs loop read /sys/devices/system/cpu<N>/topology/thread_siblings (missing the /cpu path component), never opened a file, and always fell back to half of std::thread::hardware_concurrency(). That halves every core count on a host without SMT and ignores taskset / container cpusets (a 8-CPU cpuset on a 192-CPU host ran 96 threads). Count the distinct thread-sibling sets among the CPUs in the process affinity mask instead, falling back to the old estimate over the allowed CPUs when sysfs has no topology.
…ical) The CPU CONV_2D / CONV_3D kernels (--conv-direct) gathered each patch element by element across all input planes, and wrote each GEMM output element into a different output plane. Process up to 64 consecutive patches of one output row at a time instead: each source row segment is read once per run and each output plane gets a contiguous row segment. Same values and the same fp32 -> fp16 conversions in the same slots, so results are bit-identical to the per-patch loops.
… AVX-VNNI 0008 dropped the AVX512 variants on every CPU without AVX-VNNI because Zen 4 ran them 15 to 50% slower than AVX2. That also took them from Intel Skylake-SP, Cascade Lake and Ice Lake, which run them much faster: on a 6-core Cascade Lake VM the cascadelake variant (AVX512-VNNI) cut Z-Image Turbo sampling from 50.2 s to 29.3 s and FLUX.2 klein from 63.8 s to 39.3 s against haswell. Only AMD without AVX-VNNI (Zen 4) keeps the AVX2 variant now.
…fer) --conv-direct auto now picks the direct CONV_2D / CONV_3D kernels on the CPU backend. ggml-cpu computes them with the same fp16 patches and the same GEMM as im2col + mul_mat, so outputs are bit-identical, but through a fixed 16 MB patch buffer instead of the whole im2col matrix (2.8 GB for one 128-channel conv of a 1024x1024 VAE decode, 5.6 GB for a 256-channel one). With the row-run fill from the previous commit they are also faster than im2col. Direct convolution is only taken for F16 / F32 weights, which is all the direct kernels accept; any other weight type keeps im2col instead of aborting in ggml-cpu (also for an explicit --conv-direct on).
… mask - ggml-patches/0014 also defines GGML_CPU_CONV_DIRECT_ROW_RUNS in ggml-cpu.h. CPU auto selects direct convolution only when it is defined, so a source build without the fork's ggml patches keeps the old off default instead of the slower per-element direct loops. - The comment and docs now say bit-identical holds for F16 conv weights (every Conv2d / Conv3d block); F32 LoRA tensors and depthwise convs keep F32 activations in the direct kernels and come out closer to exact. - sd_get_num_physical_cores sizes the affinity mask with CPU_ALLOC and grows it on EINVAL, for kernels built for more than 1024 CPUs. - docs/backend.md and docs/performance.md describe the CPU auto rule.
…ggml-patches/0017) (cherry picked from commit f627101079f843f63f16e3fa9a873ddb26cb3d04)
…_SGEMM_TYPES=q8_0 restores Q8_0 only)
…tches/0018) Ports the classic iqk GEMM kernels of ik_llama.cpp (commit 009dce19, MIT, Iwan Kawrakow) into ggml-cpu as src/ggml-cpu/iqk/: Q2_K, Q3_K, Q4_K, Q5_K, Q6_K, IQ4_XS, IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS, IQ3_S, IQ1_S and IQ1_M weights times Q8_K activations, for AVX2, AVX512 (F/BW/DQ/VL/VNNI) and NEON with dotprod. They read the upstream block layouts and the Q8_K rows ggml already makes, form the same integer products as vec_dot and only accumulate the floats in another order; ik's re-quantizing Q8_K_R8 conversion path is not ported. Several activation rows share each weight block decode (up to 8), instead of one row by one column per vec_dot call. Hooked in ggml_compute_forward_mul_mat after src1 is converted to Q8_K, in place of the vec_dot loop. On x86 the tiled K-quant path keeps taking ne11 >= 8 first; on ARM, which has no tiled path, these kernels take every ne11. Variants without AVX2 (or NEON dotprod) and MSVC builds compile a stub and keep vec_dot. Repacked weights do not reach this code. GGML_CPU_IQK=0 disables, GGML_CPU_IQK=2 runs them ahead of the tiled path, GGML_CPU_IQK_TYPES=q2_K,iq2_xs,... restricts the types, GGML_CPU_IQK_DEBUG=1 logs the first use per type.
…64 kernels (no lax vector conversions)
…path measured faster), on for ARM
…-aarch64) before shipping ggml dlopens every CPU variant library while scoring them, so a static initializer vectorised with a variant's AVX512 (or AVX2) flags crashes every host without that ISA. The runner's own CPU has the newest ISA and cannot see it. Replay sd-cli startup under SDE (Nehalem .. Sapphire Rapids) and qemu-aarch64 (Cortex-A53 .. max); fail on a crash, an SDE chip-check error, or a loaded variant that needs a feature the emulated CPU lacks.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
…VX2 hosts On AMD the AVX2 F16 kernels ran slower than vec_dot from the release toolchains (SD1.5 F16 sampling, GitHub runners: Zen 3 0.78x and Zen 4 0.88x with GCC 11 on ubuntu-22.04, Zen 3 0.71x with MSVC), so F16 / BF16 stay opt-in there and in every MSVC build. Q8_0 stays on everywhere. GGML_CPU_SGEMM_TYPES still overrides the set.
The Windows x64 bundle's AVX512 variants measured 1.88x on a Xeon 8573C (SD1.5 F16 sampling), so the MSVC exclusion now covers only the AVX2 kernels, where AMD Zen 3 / Zen 5 measured 0.71x / 0.86x.
…/ V2) On a Neoverse N2 (GitHub ubuntu-24.04-arm, 4 cores) the default armv8.6_2 variant rendered SD1.5 at 256 px 1.29-1.38x slower than armv8.2_2 (F16 16.1 vs 12.0 s, Q8_0 13.6 vs 9.8 s, Q3_K 18.2 vs 14.0 s, Q4_K 14.9 vs 11.2 s, two runners), and armv8.2_3 / armv8.6_1 about 5x slower. With a 128-bit SVE vector length the SVE variants now score 0 and the best NEON variant is picked; GGML_CPU_ARM_SVE128=1 restores them.
Adds Intel AVX2 under MSVC (1.24-1.25x), AMD Zen 4 AVX2 under MSVC (1.03-1.11x), AMD Zen 5 AVX2 under GCC / clang (1.07-1.08x) and aarch64 builds without FP16 vector arithmetic, whose tinyBLAS kernel accumulates in F32 (armv8.0 1.35x on a Neoverse N2; tinyBLAS is now built on aarch64 too). Stays off on Zen 3 AVX2 (0.71-0.78x), Zen 4 AVX2 with GCC 11 (0.87-0.88x), Zen 5 AVX2 with MSVC (0.86x) and aarch64 FP16-vector builds (FP16 accumulation). Q8_0 stays on x86 only.
test-backend-ops MUL_MAT perf (m 4096, k 14336) on Zen 3, Zen 4, Zen 5, Xeon 8370C / 8573C and Core i7-8700B: iqk is 1.07-3.2x faster than vec_dot for 2-7 src1 rows and 1.15-2.9x faster than the tiled path at 8 rows (not iq2_xxs / iq3_s, 0.84x / 0.91x on Zen 5), and 1.01-2.3x for K-quants up to 12 rows; tiled wins from 16 rows up and vec_dot at 1 row. x86 now takes iqk exactly there when GGML_CPU_IQK is unset; GGML_CPU_IQK=0 / 1 / 2 keep their meaning.
Adds qemu neoverse-v1 / max with a 16-byte SVE vector length: ggml-patches/0020 must pick armv8.2_2 there, and an SVE variant again with GGML_CPU_ARM_SVE128=1. Fails when a variant that has the iqk GEMM kernels built does not enable them by default (ARM, and x86 AVX2 and up since ggml-patches/0018 picks them by batch size).
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A new step on the Linux x86_64 and Linux aarch64 CPU legs of the prebuilt workflow runs the packaged bundle on emulated CPUs that the build runner is not:
sve-default-vector-length=16, the Neoverse N2 / V2 class), the second time also withGGML_CPU_ARM_SVE128=1For each CPU it starts
sd-clion a missing model, which loads and scores every CPU variant library, initialises the chosen one and stops. The step fails if the process dies, if SDE reports an instruction the emulated chip does not have, if no variant is loaded, or if the loaded variant needs a feature the emulated CPU lacks. It also checks two defaults from the CPU stack:armv8.2_2(ggml-patches/0020skips the SVE variants there), and an SVE variant whenGGML_CPU_ARM_SVE128=1;GGML_CPU_IQK_DEBUG=1prints built and enabled type masks; ARM every batch size, x86 AVX2 and up 2-12 src1 rows sinceggml-patches/0018). Pre-AVX2 x86 and Armv8.0 variants have no kernels and report0x0.This branch has the CPU stack's tip (#48, which contains #36, #42 and #47) merged in, so those expectations hold on it. Each run takes about a second, so the whole step adds roughly 20 s on x86_64 and a few seconds on aarch64.
Why
ggml dlopens every
libggml-cpu-<variant>.sowhile it scores them, so a static initializer that the compiler vectorised with one variant's flags runs on every host. If that variant is AVX512 (or SVE), the binary crashes before any variant is chosen on every CPU without that ISA. The existing CPU variants gate cannot see this: it runs on the runner's own CPU, which has the newest ISA in the bundle. A regression of this kind would ship green.Proof
Staging runs of the Linux CPU legs exactly as this workflow builds them (
-DGGML_NATIVE=OFF -DGGML_BACKEND_DL=ON -DGGML_CPU_ALL_VARIANTS=ON, same steps, other legs trimmed):iq1s_grid_usfilled by a namespace-scope initializer again (the bug fixed before #48 was opened)With the injected initializer, the existing CPU variants gate still passed on both legs (the runners loaded
zen4andarmv8.6_2), while this step failed on every CPU below the variant that carries it:Variants picked on the passing trees:
The Skylake-SP to Ice Lake rows are the intended #36 change (AVX512 variants on Intel CPUs without AVX-VNNI); every pick is within the emulated CPU's features.
After merging #48's tip (
e3c2fbd, with #36's0020) and adding the two checks: https://github.com/danielhanchen/sd-cpp-staging/actions/runs/42071800322, both legs pass.GGML_CPU_ARM_SVE128=1The earlier #48 tip (
0ad6060, run 42008423845) reportedenabled 0x0on the x86 AVX2 rows, which the new iqk check now rejects. Master prints no iqk line, so the check does not apply there.Not covered
ggml-patches/0008only removes variants.0020/ the 0018 default was put through this step.