Skip to content

perf(build): compile the C++ module in parallel - #1118

Open
ColinLeeo wants to merge 2 commits into
apache:developfrom
ColinLeeo:colin/cpp-parallel-build
Open

ColinLeeo wants to merge 2 commits into
apache:developfrom
ColinLeeo:colin/cpp-parallel-build

Conversation

@ColinLeeo

Copy link
Copy Markdown
Contributor

Summary

  • cmake --build ran without a jobs count, so the C++ module compiled one translation unit at a time under Maven. The CMAKE_BUILD_PARALLEL_LEVEL cache variable in cpp/CMakeLists.txt never had an effect: cmake --build only reads the environment variable of that name.
  • Maven now passes --parallel: mvnw/mvnw.cmd export CMAKE_BUILD_PARALLEL_LEVEL with the host core count, an env-activated profile forwards it to cmake.build.parallel.level, a plain mvn falls back to 4 jobs, and -Dcmake.build.parallel.level=N overrides both.
  • cpp/build.sh gains -j=<n> / --jobs=<n> and defaults to the detected core count; the inert CMakeLists block is removed.

Why it matters

In the unit-test (ubuntu-24.04, Release, Asan) job, compilation takes ~62 of the ~69 minutes (371 translation units, one at a time on a 4 vCPU runner), while the 1625 tests take ~6 minutes and are already parallel through ctest. This change lets the compile step use all available cores.

Notes

  • Java compilation stays single-threaded (in-process javac) and the reactor has no -T; C++ tests were already parallel.
  • No codec, file-format, or runtime behavior changes.

Test plan

  • ./mvnw -P with-cpp clean test-compile on a 15-core host logs cmake --build ... --parallel 15; clean compile 1:48.
  • ./mvnw clean verify -P with-java,with-python -Denable.lzma2=ON -> BUILD SUCCESS (258 Java tests, 1624 ctest cases, Python pytest).
  • ASan: DEVELOPER_DIR=<Xcode 26> ASAN_OPTIONS=detect_container_overflow=0 ./mvnw clean verify -P with-cpp -Denable.asan=ON -Dbuild.type=RelWithDebInfo -Denable.lzma2=ON -> 1625/1625 tests passed.
  • ./mvnw spotless:check -P with-java,with-python -> BUILD SUCCESS.

cmake --build runs the native tool one file at a time unless a jobs count is
given, and the CMAKE_BUILD_PARALLEL_LEVEL cache variable in cpp/CMakeLists.txt
had no effect: only the environment variable of that name is read. Every Maven
C++ build therefore compiled serially, which dominates CI time in the
Release + ASan job.

Pass --parallel to the compile execution and take the level from
CMAKE_BUILD_PARALLEL_LEVEL, which ./mvnw and mvnw.cmd now export with the host
core count. A plain mvn falls back to 4 jobs, and -Dcmake.build.parallel.level=N
overrides both. build.sh gains -j/--jobs for the same default, and the inert
CMakeLists block is removed.
@ColinLeeo
ColinLeeo requested a balanced review from Copilot November 16, 2026 07:13

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

3 open issues
What changed in this PR

Improve native (C++) build throughput by enabling parallel compilation across Maven and cpp/build.sh, fixing the previously ineffective CMake cache-based parallelism.

Changes:

  • Export CMAKE_BUILD_PARALLEL_LEVEL in mvnw/mvnw.cmd and wire it into Maven via an env-activated profile with a 4-job fallback.
  • Pass explicit parallelism to the native build (cmake --build ... --parallel N in Maven; make -jN in cpp/build.sh).
  • Update docs/READMEs and remove the inert CMAKE_BUILD_PARALLEL_LEVEL cache-variable logic from cpp/CMakeLists.txt.
File Description
skills/​tsfile/​references/​cpp-build-write.md Documents new parallel build behavior and configuration knobs.
mvnw.cmd Sets CMAKE_BUILD_PARALLEL_LEVEL on Windows wrapper runs.
mvnw Detects core count and exports CMAKE_BUILD_PARALLEL_LEVEL in the Unix wrapper.
cpp/​pom.xml Adds --parallel <N> to CMake build and wires env var into a Maven property.
cpp/​build.sh Adds --jobs/-j and defaults to detected core count for make -j.
cpp/​README.md Documents Maven/build.sh parallel compilation behavior.
cpp/​README-zh.md Documents parallel compilation behavior in Chinese README.
cpp/​CMakeLists.txt Removes ineffective cache-variable parallelism; adds explanatory comment.
cpp/​CLAUDE.md Documents the new parallelism behavior and overrides.

Review effort: Lite

Comment thread mvnw Resolved
Comment thread mvnw.cmd Resolved
Comment thread cpp/build.sh Resolved
Verify that each core-count probe actually returns a value before trusting
it, and add a getconf _NPROCESSORS_ONLN fallback: sysctl exists on Linux but
hw.ncpu does not. Keep CMAKE_BUILD_PARALLEL_LEVEL undefined on Windows when
NUMBER_OF_PROCESSORS is missing, and accept -j<n> in build.sh.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants