ci: add release workflow to publish prebuilt SDK binaries - #628
Open
chinaux wants to merge 27 commits into
Open
Conversation
chinaux
force-pushed
the
feat/release-prebuilt
branch
2 times, most recently
from
August 12, 2026 03:10
a53a9d0 to
2b45a34
Compare
zhourrr
requested changes
Aug 12, 2026
chinaux
force-pushed
the
feat/release-prebuilt
branch
3 times, most recently
from
August 18, 2026 11:38
79914e4 to
ea0c836
Compare
chinaux
added a commit
to chinaux/zvec
that referenced
this pull request
Aug 18, 2026
The wait_for predicates introduced by alibaba#680 reference the function-local constexpr kTaskCount without capturing it. GCC/Clang treat the immediate read of a constant expression as non-odr-use and accept it, but MSVC rejects it with C3493, breaking the windows-2025 build (seen in PR alibaba#628 CI; main's Windows job is skipped so it went unnoticed). Capture the constant explicitly.
chinaux
force-pushed
the
feat/release-prebuilt
branch
from
August 20, 2026 02:47
2eb0cd2 to
1bb5194
Compare
zhourrr
reviewed
Aug 20, 2026
- release.yml: triggered by v* tag push or manual dispatch, creates GitHub Release via softprops/action-gh-release - _build_prebuilt.yml: reusable workflow building zvec SDK (include/ + lib/, shared libs only) for osx-arm64, linux-amd64, linux-arm64 and windows-amd64 with DuckDB-style asset naming
…lock When called via workflow_call, github.workflow resolves to the caller's name, making the reusable workflow compete with the caller for the same concurrency group and deadlocking the run. Concurrency is controlled by the caller (release.yml).
lz4's Makefile.inc defines 'INSTALL_DIR ?= install -d -m 755' as a command, which gets overridden by our INSTALL_DIR environment variable, breaking 'make install' on all Unix platforms. Prefix our env vars with ZVEC_.
Linux prebuilt .so files were linked against glibc 2.38 (ubuntu-24.04), making them incompatible with older distros. Run the Linux build jobs inside manylinux_2_28 containers (glibc 2.28) to match the Python wheel build's compatibility — see [tool.cibuildwheel.linux] in pyproject.toml. - Add per-matrix 'container' field; macOS/Windows use '' (no container) - Skip actions/setup-python on Linux (container provides python3+pip) - Change 'python' to 'python3' in Unix install step for container compat Android and iOS jobs are unchanged (cross-compiled, no host glibc dep).
The manylinux_2_28 container's system Python (/usr/bin/python3) does not have pip installed. Use the manylinux-bundled CPython at /opt/python/cp311/bin/ instead, which includes pip. Fall back to python3 from actions/setup-python on non-Linux platforms.
The manylinux_2_28 image does not have /opt/python/cp311/bin (the previous fallback path). Use actions/setup-python on all platforms including inside the Linux container — it installs Python 3.11 with pip into the tool cache which is mounted into the container. Also remove pip caching to avoid container cache path issues.
actions/setup-python downloads a CPython built for glibc 2.34+, which cannot run inside the manylinux_2_28 container (glibc 2.28). Use the manylinux-bundled CPython at /opt/python/cp3*/bin/ instead, which has pip and is compatible with the container's glibc. - Skip actions/setup-python on Linux (glibc incompatibility) - Glob /opt/python/cp3*/bin to find any bundled CPython version - Add to GITHUB_PATH so cmake/ninja are available in later steps - Fall back to python3 from setup-python on macOS
The shell uses 'bash -e -o pipefail'. On macOS, the glob /opt/python/cp3*/bin doesn't match, causing 'ls' to fail and pipefail to propagate the non-zero exit, triggering set -e and killing the script before reaching pip install. Add '|| true' to the command substitution to prevent this.
The prebuilt SDK matrix only covered glibc Linux (manylinux_2_28), while musl was validated by 09-musllinux-build.yml and the musllinux wheels without ever producing downloadable artifacts.
- Add a linux-musl-amd64 matrix entry using the pinned quay.io/pypa/musllinux_1_2_x86_64 image (same as 09-musllinux-build.yml and [tool.cibuildwheel.linux] in pyproject.toml), plus an ensure-bash step since the Alpine-based image may not ship the bash used by the shared steps.
- Add a dedicated build-musl-arm64 job: JavaScript actions cannot run inside Alpine containers on arm64 runners, so it drives the container via docker exec, mirroring build-and-test-arm64 in 09-musllinux-build.yml.
release.yml picks up zvec-sdk-linux-musl-{amd64,arm64} automatically via its zvec-sdk-* download pattern.
The musl prebuilt builds have been validated via the v0.0.0-musl-test-rc.1 tag run, so the temporary feat/release-prebuilt push trigger is no longer needed. The workflow now runs only via workflow_call (release.yml) or manual dispatch.
…ude/ install(DIRECTORY <dir> DESTINATION include) without a trailing slash copies the directory itself, so the zvec PACKED public headers landed at <prefix>/include/include/zvec/ while the C API header landed at <prefix>/include/zvec/c_api.h. The nested path also contradicts the INSTALL_INTERFACE:include usage requirement of the all-in-one targets. Use a trailing slash so the header tree is installed as <prefix>/include/zvec/.
Previously the prebuilt jobs packaged and uploaded the SDK without verifying it. Each platform job now checks the staged package before packaging: - Desktop jobs (glibc/musl Linux x64+arm64, macOS, Windows) compile and run a tiny C program (links libzvec_c_api, calls the version API) and a tiny C++ program (links the all-in-one zvec library, calls zvec::GetDefaultMessage) against the staged include/ + lib/ tree. This catches missing files, header-install-layout regressions, and unresolvable symbols or load failures. - Android cross-compiles the same programs against the packaged .so files (link-only; arm64 binaries cannot run on the runner). - iOS verifies every XCFramework carries both device and simulator slices and that the headers are present at include/zvec/. - The shared smoke project lives in .github/cmake/sdk-smoke/ and fails configuration with a clear message if the package is incomplete.
The NDK toolchain re-roots find_library searches (CMAKE_FIND_ROOT_PATH_MODE_LIBRARY), so the explicit PATHS into the staged SDK were ignored and the Android smoke step failed to configure. Add CMAKE_FIND_ROOT_PATH_BOTH to search the package path as given.
alibaba#627 removed the protobuf dependency, so there is no protoc target to build for host code generation anymore. Mirror the updated scripts/build_android.sh and scripts/build_ios.sh: - Drop the host protoc build, the thirdparty reset that prepared for it, and the GLOBAL_CC_PROTOBUF_PROTOC configure flag from both the android and ios jobs. - Switch the Android build to ANDROID_STL=c++_static and stop bundling libc++_shared.so, matching the slimmed self-contained libraries.
The mutable :latest tag of quay.io/pypa/manylinux_2_28_* failed to resolve in CI (see review screenshot), which makes the glibc SDK builds depend on quay.io tag availability at run time. Pin 2026.03.01-1, the same image batch as the musllinux pins.
release.yml triggered _build_prebuilt.yml without forwarding the release tag. The reusable workflow checks out shallow and tag-less, so cmake/version.cmake fell back to the dummy v0.0.0 (see the 'zvec version: v0.0.0' warning in prebuilt release logs). Add a 'tag' input to _build_prebuilt.yml (workflow_call + workflow_dispatch) and expand it to -DOVERRIDE_GIT_DESCRIBE=<tag> via the ZVEC_VERSION_DEFINE env var, appended to every library configure step (matrix, musl-arm64, android, ios). release.yml now passes github.ref_name on tag pushes and the dispatch tag input otherwise. Ad-hoc runs without a tag keep the v0.0.0 fallback.
Spot-check the linux-amd64 SDK in the release job: the version string resolved by cmake/version.cmake is compiled into libzvec_c_api.so (zvec_get_version), so the exact release tag must appear in the binary. Fail the release if it does not match, so a missing tag input (silent v0.0.0 fallback) can never be published unnoticed. Validated against a real prebuilt artifact: passes with the matching tag and fails with an actionable ::error otherwise.
- Validate that all 8 platform SDK archives are present before creating the release, so an incomplete build set fails loudly instead of shipping silently. - Pass the expected tag to the version spot-check via step env instead of inline expression expansion in the shell script.
The reusable workflow checked out the inherited event ref, which is only correct for tag-push runs. A manual release dispatch made after the branch moved past the tag would build branch HEAD while stamping the tag's version. Derive ZVEC_CHECKOUT_REF from the tag input (falling back to the triggering ref for ad-hoc runs) and pass it to every checkout step, so the built artifacts always correspond exactly to the released tag.
The jieba tokenizer is the only FTS tokenizer that needs external data (jieba.dict.utf8 + hmm_model.utf8, covering all four cut modes). The Python wheel already bundles them (zvec/data/jieba_dict, auto-registered on import), but prebuilt C/C++ SDK packages shipped nothing, forcing consumers to source cppjieba's dict themselves and risk version skew. - Install the two dict files to <prefix>/share/zvec/jieba_dict on every native SDK install (default component; the COMPONENT-python wheel build is unaffected). Reuses the same thirdparty/cppjieba source as the wheel, so both artifacts stay in sync. - sdk-smoke now fails if the staged package lacks the dict files. - iOS packaging copies share/ from the device slice and verifies the dict. - The release job spot-checks the dict files in the linux-amd64 SDK. C/C++ consumers register the directory via zvec_set_default_jieba_dict_dir(), zvec_config_data_set_jieba_dict_dir(), ZVEC_JIEBA_DICT_DIR, or per-field extra_params.jieba_dict_dir (priority: per-field > env > global default).
chinaux
force-pushed
the
feat/release-prebuilt
branch
from
August 21, 2026 10:24
f7f1a9d to
9cb776d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds a GitHub Release pipeline that publishes prebuilt C++/C SDK binaries, so users can consume zvec directly without building from source.
What's included
New workflows
.github/workflows/release.yml— the release entry point:v*tag, or manually viaworkflow_dispatchwith a tag inputauto-generated notes via
softprops/action-gh-releasealpha/beta/rcare automatically marked as prerelease.github/workflows/_build_prebuilt.yml— reusable build workflow(
workflow_call+workflow_dispatch), building the SDK for all platformtargets in parallel
Release assets
zvec-sdk-linux-amd64.tar.gzubuntu-24.04manylinux_2_28container (glibc ≥ 2.28, e.g. CentOS 8 / Ubuntu 20.04+)zvec-sdk-linux-arm64.tar.gzubuntu-24.04-armmanylinux_2_28containerzvec-sdk-linux-musl-amd64.tar.gzubuntu-24.04musllinux_1_2container (Alpine/musl), same image as the musllinux wheel buildszvec-sdk-linux-musl-arm64.tar.gzubuntu-24.04-armmusllinux_1_2container driven viadocker exec(JS actions don't run in Alpine containers on arm64 runners)zvec-sdk-osx-arm64.tar.gzmacos-15zvec-sdk-windows-amd64.zipwindows-2025zvec-sdk-android-arm64.tar.gzubuntu-24.04c++_static(self-contained since #627)zvec-sdk-ios.zipmacos-15cmake fixes needed for the SDK builds
cmake/bazel.cmake: on iOS the<name>and<name>_statictargets now share a single archive (alias), avoiding Ninja "multiple rules generate" failures and duplicate-symbol links<prefix>/include/zvec/instead of the nested<prefix>/include/include/zvec/Artifact verification before upload
Every build job verifies the staged package before packaging it, using the shared smoke project in
.github/cmake/sdk-smoke/:libzvec_c_api, calls the version API) and a small C++ program (links the all-in-onezveclibrary, callszvec::GetDefaultMessage) against the stagedinclude/+lib/tree. This catches missing files, header-install-layout regressions, unresolvable symbols, and load failures..sofiles (link-only; arm64 binaries cannot execute on the runner).ios-arm64andios-arm64-simulatorslices and that the headers are present atinclude/zvec/.Configuration fails with a clear message if the package is incomplete (e.g., missing
include/zvec/c_api.h).