Enhance macOS compatibility with spawn-based multiprocessing#4883
Open
gamesh411 wants to merge 6 commits into
Open
Enhance macOS compatibility with spawn-based multiprocessing#4883gamesh411 wants to merge 6 commits into
gamesh411 wants to merge 6 commits into
Conversation
fc3d34d to
976efeb
Compare
ee2abfd to
09c136a
Compare
6b66186 to
d77eec4
Compare
8 tasks
42d1258 to
a7513ea
Compare
a7513ea to
b25924a
Compare
38b44c8 to
e22d92e
Compare
Replace the custom compatibility.multiprocessing shim with direct imports from the multiprocess package. On macOS and Windows, the default fork start method is unsafe (Obj-C runtime crashes) or unavailable, so set spawn explicitly in cli.py:main(). Spawn workers do not inherit parent state, so analysis_manager and pre_analysis_manager now explicitly set up logging in spawned workers, and log_parser's _init_log_parser_worker propagates the shared compiler_info dict to children. Add mypy override for multiprocess (no type stubs available).
On macOS/Windows (spawn), worker processes cannot inherit the HTTPServer instance or shared Queue/Value objects from the parent. Reconstruct the server in spawned API workers from serializable config (_build_worker_server) and use SyncManager.Queue() for the task queue so it can be pickled to spawned task workers. Spawned workers detect shutdown via SIGINT (not a shared flag), and the cwd is validated before spawning to avoid os.getcwd() failures if the inherited cwd was deleted.
The JSON compilation database spec allows entries to use either 'command' (string) or 'arguments' (list). The intercept-build wrapper used on macOS (bear) produces the 'arguments' form. Normalize 'arguments' to 'command' in tu_collector wherever the build action command is accessed, so both formats work on all platforms.
Add CC_TEST_API_WORKERS and CC_TEST_TASK_WORKERS env vars that map to --api-handler-processes and --task-worker-processes server flags. On macOS CI, spawn workers are expensive (each re-imports the full Python stack), so constraining to 1 worker reduces resource pressure. Also improve wait_for_server_start with an HTTP fallback check (port-based readiness) and fail-fast diagnostics for common startup errors.
Analyzer tests:
- Use os.path.realpath('/tmp') to handle macOS /tmp -> /private/tmp
symlink in log_parser and option_parser tests
- Skip LD_PRELOAD/LD_LIBRARY_PATH tests on macOS (not applicable)
- Skip gcc analyzer test when g++ is an Apple clang shim
- Skip -m32 test on macOS (not available)
- Use -fsyntax-only in buildcmd escaping tests (linking may fail
with Homebrew clang on macOS)
- Normalize intercept-build vs ld-logger output strings
- Normalize gcc/infer diagnostic quoting and type annotations
across versions for cross-platform comparison
- Add self.fail() in test_skip for tu_collector errors
- Guard 'command' key access for arguments-format entries
Web tests:
- OAuth mock server: bind to 127.0.0.1, poll for readiness
instead of fixed sleep, track and terminate the process in
teardown, use sys.executable instead of 'python3'
- blame tests: restore cwd in finally block to prevent test
pollution on assertion failure
- task_management tests: poll for state transitions instead of
fixed sleeps (spawn workers take ~42s to start on macOS),
use 5s tasks to reliably observe RUNNING state
- Makefile: use $(ROOT)/build for workspace and CodeChecker cmd
paths (consistent with __init__.py's hardcoded PKG_ROOT)
- env.py: no functional change
Add install-deps-macos.sh that installs llvm@14, gcc@13, cppcheck, bear, and Infer (from GitHub releases, no Homebrew formula). The script creates an intercept-build wrapper around bear (LLVM's libear.dylib is broken on macOS ARM64 with SIP), gcc/g++ symlinks matching Linux naming, and sets SDKROOT/MACOSX_DEPLOYMENT_TARGET/ CC/CXX to ensure correct compilation and SARIF output. Add macOS matrix entries to the analyzer and tools jobs, and a separate web-macos job (sqlite only, with reduced worker counts via CC_TEST_API_WORKERS/CC_TEST_TASK_WORKERS). All macOS jobs use continue-on-error as they are experimental.
e22d92e to
64a46d1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Enables the full analyzer and web test suites to run on macOS by switching from fork to spawn as the multiprocessing start method on non-Linux platforms, and making the test suite cross-platform.
CodeChecker currently only runs CI on Linux. macOS uses fork by default but it is unsafe (Obj-C runtime crashes in child processes, see #4893 ), and the test suite has several Linux-specific assumptions (/tmp path, LD_PRELOAD, ld_logger, compile_commands.json command format, fixed time.sleep for task state transitions). This PR addresses all of these so the existing test suite passes on macOS with no behavioral changes on Linux.
Changes
Implementation
Remove compatibility.multiprocessing shim, use multiprocess directly.
The custom compatibility wrapper is replaced with direct imports from the multiprocess package.
The spawn start method is set explicitly in
cli.py:main()on macOS/Windows.Spawn workers get explicit logger setup and shared state propagation (compiler_info dict) since they don't inherit parent state via fork.
Spawn-based worker processes in the server.
On macOS/Windows, API and task worker processes cannot inherit the
HTTPServerinstance or sharedQueue/Valueobjects.The server is reconstructed in spawned API workers from serializable config (
_build_worker_server), andSyncManager.Queue()is used for the task queue so it can be pickled to spawned task workers.arguments list format in compile_commands.json.
The JSON compilation database spec allows entries to use either command (string) or arguments (list).
The intercept-build wrapper used on macOS (via bear) produces the arguments form. tu_collector now normalizes both formats wherever the build action command is accessed.
CI worker count override.
CC_TEST_API_WORKERSandCC_TEST_TASK_WORKERSenv vars map to--api-handler-processesand--task-worker-processesserver flags.On macOS CI, spawn workers are expensive (each re-imports the full Python stack), so constraining to 1 worker reduces resource pressure.
Also improves wait_for_server_start with an HTTP port-based fallback and fail-fast diagnostics.
Test infra
os.path.realpath('/tmp')to handle macOS/tmpbeing a symlink to/private/tmpLD_PRELOAD/LD_LIBRARY_PATH/-m32tests on macOS (not applicable)-fsyntax-onlyin buildcmd escaping tests (linking may fail with Homebrew clang@14)track and terminate in teardown
(spawn workers take ~42s to start), use 5s tasks to reliably observe RUNNING state
GH Actions
Creates an intercept-build wrapper around bear (LLVM's libear.dylib seems broken on macOS ARM64 with SIP),
gcc/g++ symlinks matching Linux naming,
and sets SDKROOT/MACOSX_DEPLOYMENT_TARGET/CC/CXX for correct compilation and SARIF output.
separate web-macos job (sqlite only, reduced worker counts). All macOS jobs use continue-on-error.