Skip to content

Enhance macOS compatibility with spawn-based multiprocessing#4883

Open
gamesh411 wants to merge 6 commits into
Ericsson:masterfrom
gamesh411:macos-test-support
Open

Enhance macOS compatibility with spawn-based multiprocessing#4883
gamesh411 wants to merge 6 commits into
Ericsson:masterfrom
gamesh411:macos-test-support

Conversation

@gamesh411

@gamesh411 gamesh411 commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Enables the full analyzer and web test suites to run on macOS by switching from fork to spawn as the multiprocessing start method on non-Linux platforms, and making the test suite cross-platform.

CodeChecker currently only runs CI on Linux. macOS uses fork by default but it is unsafe (Obj-C runtime crashes in child processes, see #4893 ), and the test suite has several Linux-specific assumptions (/tmp path, LD_PRELOAD, ld_logger, compile_commands.json command format, fixed time.sleep for task state transitions). This PR addresses all of these so the existing test suite passes on macOS with no behavioral changes on Linux.

Changes

Implementation

  1. Remove compatibility.multiprocessing shim, use multiprocess directly.
    The custom compatibility wrapper is replaced with direct imports from the multiprocess package.
    The spawn start method is set explicitly in cli.py:main() on macOS/Windows.
    Spawn workers get explicit logger setup and shared state propagation (compiler_info dict) since they don't inherit parent state via fork.

  2. Spawn-based worker processes in the server.
    On macOS/Windows, API and task worker processes cannot inherit the HTTPServer instance or shared Queue/Value objects.
    The server is reconstructed in spawned API workers from serializable config (_build_worker_server), and SyncManager.Queue() is used for the task queue so it can be pickled to spawned task workers.

  3. arguments list format in compile_commands.json.
    The JSON compilation database spec allows entries to use either command (string) or arguments (list).
    The intercept-build wrapper used on macOS (via bear) produces the arguments form. tu_collector now normalizes both formats wherever the build action command is accessed.

  4. CI worker count override.
    CC_TEST_API_WORKERS and CC_TEST_TASK_WORKERS env vars map to --api-handler-processes and --task-worker-processes server flags.
    On macOS CI, spawn workers are expensive (each re-imports the full Python stack), so constraining to 1 worker reduces resource pressure.
    Also improves wait_for_server_start with an HTTP port-based fallback and fail-fast diagnostics.

Test infra

  • os.path.realpath('/tmp') to handle macOS /tmp being a symlink to /private/tmp
  • Skip LD_PRELOAD/LD_LIBRARY_PATH/-m32 tests on macOS (not applicable)
  • Skip gcc analyzer test when g++ is an Apple clang shim
  • -fsyntax-only in buildcmd escaping tests (linking may fail with Homebrew clang@14)
  • Normalize intercept-build vs ld-logger output strings for cross-platform comparison
  • Normalize gcc/infer diagnostic quoting and type annotations across versions
  • OAuth mock server: bind to 127.0.0.1, poll for readiness instead of fixed sleep,
    track and terminate in teardown
  • Blame tests: restore cwd in finally block to prevent test pollution
  • Task management tests: poll for state transitions instead of fixed sleeps
    (spawn workers take ~42s to start), use 5s tasks to reliably observe RUNNING state

GH Actions

  • install-deps-macos.sh: installs llvm@14, gcc@13, cppcheck, bear, and Infer.
    Creates an intercept-build wrapper around bear (LLVM's libear.dylib seems broken on macOS ARM64 with SIP),
    gcc/g++ symlinks matching Linux naming,
    and sets SDKROOT/MACOSX_DEPLOYMENT_TARGET/CC/CXX for correct compilation and SARIF output.
  • test.yml: macOS matrix entries for analyzer and tools jobs,
    separate web-macos job (sqlite only, reduced worker counts). All macOS jobs use continue-on-error.

@gamesh411 gamesh411 force-pushed the macos-test-support branch from fc3d34d to 976efeb Compare June 8, 2026 13:10
@gamesh411 gamesh411 changed the title [test][ci] Macos test support [test][ci] MacOS test support Jun 8, 2026
@gamesh411 gamesh411 force-pushed the macos-test-support branch 28 times, most recently from ee2abfd to 09c136a Compare June 11, 2026 18:30
@gamesh411 gamesh411 force-pushed the macos-test-support branch 7 times, most recently from 6b66186 to d77eec4 Compare June 12, 2026 14:08
@gamesh411 gamesh411 force-pushed the macos-test-support branch 2 times, most recently from 42d1258 to a7513ea Compare June 12, 2026 20:54
@gamesh411 gamesh411 force-pushed the macos-test-support branch from a7513ea to b25924a Compare June 30, 2026 20:45
@gamesh411 gamesh411 force-pushed the macos-test-support branch 5 times, most recently from 38b44c8 to e22d92e Compare July 1, 2026 13:42
gamesh411 added 6 commits July 1, 2026 20:04
Replace the custom compatibility.multiprocessing shim with direct
imports from the multiprocess package. On macOS and Windows, the
default fork start method is unsafe (Obj-C runtime crashes) or
unavailable, so set spawn explicitly in cli.py:main().

Spawn workers do not inherit parent state, so analysis_manager and
pre_analysis_manager now explicitly set up logging in spawned
workers, and log_parser's _init_log_parser_worker propagates the
shared compiler_info dict to children.

Add mypy override for multiprocess (no type stubs available).
On macOS/Windows (spawn), worker processes cannot inherit the
HTTPServer instance or shared Queue/Value objects from the parent.
Reconstruct the server in spawned API workers from serializable
config (_build_worker_server) and use SyncManager.Queue() for the
task queue so it can be pickled to spawned task workers.

Spawned workers detect shutdown via SIGINT (not a shared flag),
and the cwd is validated before spawning to avoid os.getcwd()
failures if the inherited cwd was deleted.
The JSON compilation database spec allows entries to use either
'command' (string) or 'arguments' (list). The intercept-build
wrapper used on macOS (bear) produces the 'arguments' form.

Normalize 'arguments' to 'command' in tu_collector wherever the
build action command is accessed, so both formats work on all
platforms.
Add CC_TEST_API_WORKERS and CC_TEST_TASK_WORKERS env vars that
map to --api-handler-processes and --task-worker-processes server
flags. On macOS CI, spawn workers are expensive (each re-imports
the full Python stack), so constraining to 1 worker reduces
resource pressure.

Also improve wait_for_server_start with an HTTP fallback check
(port-based readiness) and fail-fast diagnostics for common
startup errors.
Analyzer tests:
- Use os.path.realpath('/tmp') to handle macOS /tmp -> /private/tmp
  symlink in log_parser and option_parser tests
- Skip LD_PRELOAD/LD_LIBRARY_PATH tests on macOS (not applicable)
- Skip gcc analyzer test when g++ is an Apple clang shim
- Skip -m32 test on macOS (not available)
- Use -fsyntax-only in buildcmd escaping tests (linking may fail
  with Homebrew clang on macOS)
- Normalize intercept-build vs ld-logger output strings
- Normalize gcc/infer diagnostic quoting and type annotations
  across versions for cross-platform comparison
- Add self.fail() in test_skip for tu_collector errors
- Guard 'command' key access for arguments-format entries

Web tests:
- OAuth mock server: bind to 127.0.0.1, poll for readiness
  instead of fixed sleep, track and terminate the process in
  teardown, use sys.executable instead of 'python3'
- blame tests: restore cwd in finally block to prevent test
  pollution on assertion failure
- task_management tests: poll for state transitions instead of
  fixed sleeps (spawn workers take ~42s to start on macOS),
  use 5s tasks to reliably observe RUNNING state
- Makefile: use $(ROOT)/build for workspace and CodeChecker cmd
  paths (consistent with __init__.py's hardcoded PKG_ROOT)
- env.py: no functional change
Add install-deps-macos.sh that installs llvm@14, gcc@13, cppcheck,
bear, and Infer (from GitHub releases, no Homebrew formula). The
script creates an intercept-build wrapper around bear (LLVM's
libear.dylib is broken on macOS ARM64 with SIP), gcc/g++ symlinks
matching Linux naming, and sets SDKROOT/MACOSX_DEPLOYMENT_TARGET/
CC/CXX to ensure correct compilation and SARIF output.

Add macOS matrix entries to the analyzer and tools jobs, and a
separate web-macos job (sqlite only, with reduced worker counts
via CC_TEST_API_WORKERS/CC_TEST_TASK_WORKERS). All macOS jobs use
continue-on-error as they are experimental.
@gamesh411 gamesh411 force-pushed the macos-test-support branch from e22d92e to 64a46d1 Compare July 1, 2026 18:08
@gamesh411 gamesh411 changed the title [test][ci] MacOS test support Add macOS test support with spawn-based multiprocessing Jul 1, 2026
@gamesh411 gamesh411 marked this pull request as ready for review July 1, 2026 19:55
@gamesh411 gamesh411 requested review from bruntib and vodorok as code owners July 1, 2026 19:55
@gamesh411 gamesh411 changed the title Add macOS test support with spawn-based multiprocessing Enhance macOS compatibility and add test support with spawn-based multiprocessing Jul 1, 2026
@gamesh411 gamesh411 changed the title Enhance macOS compatibility and add test support with spawn-based multiprocessing Enhance macOS compatibility with spawn-based multiprocessing Jul 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant