ci(macos): cache thirdparty lib/include outputs across runs by Fedr · Pull Request #6100 · MeshInspector/MeshLib

Fedr · 2026-05-14T17:15:20Z

Summary

The "Install thirdparty libs" step on macOS takes ~9 min on the x64 hosted runner: brew installs + compiling the source-built thirdparty libs (libE57Format, OpenCTM, googletest, parallel-hashmap, glad, mrbind-pybind11, tinygltf, laz-perf, clip, fastmcpp) into ./lib + ./include. The submodules feeding that compile rarely change between runs, so the work repeats unchanged on nearly every job. Reference cache-miss baseline: job 76017787493.

This PR adds actions/cache@v5 around ./lib + ./include so the compile is skipped on cache hit. (v5 to match build-mrbind, install-msys2-mrbind, and install-cuda in this repo.)

What changed in the workflow

scripts/build_thirdparty.sh is unchanged from master. All logic lives in .github/workflows/build-test-macos.yml:

New step Compute cache key inputs produces two outputs in a single bash invocation:
- brew-hash — hash of $(brew --prefix), consumed by Build MRBind below to discriminate self-hosted ARM fleet members.
- thirdparty-hash — one unified hash of everything that influences the thirdparty ./lib + ./include outputs (brew prefix + submodule SHAs from git ls-tree HEAD + contents of scripts/build_thirdparty.sh, scripts/install_brew_requirements.sh, scripts/thirdparty/{clip,fastmcpp}.sh, thirdparty/CMakeLists.txt, requirements/macos.txt).
New step Cache thirdparty build outputs with key v1-thirdparty-${matrix.instance}-${matrix.compiler}-${thirdparty-hash}.
The old Install thirdparty libs step is split into two:
- Install brew requirements — always runs, calls scripts/install_brew_requirements.sh directly. Needed on cache hit too because cached .dylibs link against brew libs at runtime.
- Build thirdparty libs — gated if: steps.thirdparty-cache.outputs.cache-hit != 'true', calls scripts/build_thirdparty.sh.

Measured impact

Step "Install thirdparty libs" duration, measured on an earlier commit of this branch that still ran the steps as a single unit:

Leg	Cache-miss	Cache-hit	Saving
x64 Release (`macos-15-intel`)	9m 14s	5m 27s	3m 47s
arm64 Debug (`macos-14`)	3m 54s	1m 53s	2m 1s
arm64 Release (self-hosted ARM)	1m 12s	31s	41s

The savings come from skipping the cmake compile portion (now the gated Build thirdparty libs step). Install brew requirements runs on both paths. The x64 hosted leg gets the biggest win because that runner image has fewer of the required bottles pre-cached. The self-hosted ARM already has brew packages persisted across runs, so its cache-miss is short to begin with; the saving there is just the cmake compile portion.

What gets cached

./lib — final .dylib / .a from cmake --install
./include — installed headers

What does not get cached:

./thirdparty_build/ — the CMake build dir containing *.o, CMakeFiles/, generated Makefiles, etc. Rebuilt from scratch on a miss and discarded on a hit.

Cache sizes observed: ~3.7-3.8 MiB per leg (confirming *.o is not included; the CMake build dir would be hundreds of MB).

Cache key

v1-thirdparty-${matrix.instance}-${matrix.compiler}-${thirdparty-hash} where thirdparty-hash is the unified hash described above. The v1- prefix is hardcoded so future incompatibilities can be busted by a one-line bump.

Test plan

First CI run: cache miss on all three matrix legs, thirdparty compile runs normally, cache gets populated (~3.7-3.8 MiB each).
Second push: cache hits on all three legs, Build thirdparty libs step skipped (if: cache-hit != 'true' false), Install brew requirements still runs.
Downstream Build / Generate ... bindings / MRMesh Exported Symbols / unit tests / pkg creation all pass on cache hit.
Cache size confirms ./thirdparty_build/*.o is not included.

build_thirdparty.sh on macOS spends ~4 min compiling the source-built thirdparty libs (libE57Format, OpenCTM, googletest, parallel-hashmap, glad, mrbind-pybind11, tinygltf, laz-perf, clip, fastmcpp) into ./lib and ./include. The submodules pinned to those SHAs rarely change between runs, so the work repeats unchanged on nearly every job. Add an actions/cache@v4 step keyed on: - matrix.instance + matrix.compiler (runner image + toolchain) - brew prefix hash (self-hosted ARM fleet may use different prefixes) - SHAs of the source-built submodules (git ls-tree HEAD) - hash of build_thirdparty.sh, install_brew_requirements.sh, thirdparty/{clip,fastmcpp}.sh, thirdparty/CMakeLists.txt, requirements/macos.txt Only ./lib and ./include get cached -- those hold the final .dylib/.a + headers produced by `cmake --install`. ./thirdparty_build (the CMake build dir holding *.o, CMakeFiles/, etc.) is NOT cached; it's rebuilt on a miss and discarded on a hit. build_thirdparty.sh gains a MESHLIB_THIRDPARTY_SKIP_BUILD=1 short-circuit that runs after install_brew_requirements.sh (so brew packages always get reinstalled on fresh hosted runners for dyld) but before the rm -rf + cmake stage. The macOS workflow sets the env var iff the cache restored. The pre-existing "Compute brew-prefix hash for MRBind cache key" step gets moved up and its `if: mrbind` guard dropped so both caches can share it; the Build MRBind reference (steps.brew-hash.outputs.hash) is unchanged. Cache key prefix `v1-` is hardcoded so future incompatibilities can be busted by a one-line bump. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Match the version already used by build-mrbind, install-msys2-mrbind, and install-cuda in this repo. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

oitel · 2026-05-15T08:20:32Z


+# CI: reuse cached ./lib and ./include from a previous run. Brew install above
+# still runs so dyld can resolve runtime deps.
+if [[ "${MESHLIB_THIRDPARTY_SKIP_BUILD:-}" == "1" ]]; then


What's the point of early-exiting within the script instead of skipping it?

oitel · 2026-05-15T08:22:12Z

+      # build_thirdparty.sh actually compiles into ./lib + ./include on macOS.
+      # `git ls-tree` reads the gitlink entries from the index, so it works
+      # whether or not the submodules are checked out.
+      - name: Compute thirdparty submodule SHA for cache key


Compute all hashes in a single step.

oitel · 2026-05-15T08:22:59Z

+      # different paths (e.g. `/Users/runner/.homebrew` vs `/opt/homebrew`) get
+      # distinct cache entries -- the prefix is baked into thirdparty rpaths
+      # and into mrbind's rpath.
+      - name: Compute brew-prefix hash for cache keys


Compute all hashes in a single step.

oitel · 2026-05-15T08:23:04Z

+          path: |
+            lib
+            include
+          key: v1-thirdparty-${{ matrix.instance }}-${{ matrix.compiler }}-${{ steps.brew-hash.outputs.hash }}-${{ steps.thirdparty-src-hash.outputs.hash }}-${{ hashFiles('scripts/build_thirdparty.sh', 'scripts/install_brew_requirements.sh', 'scripts/thirdparty/clip.sh', 'scripts/thirdparty/fastmcpp.sh', 'thirdparty/CMakeLists.txt', 'requirements/macos.txt') }}


Compute all hashes in a single step.

Address review feedback: - Restore scripts/build_thirdparty.sh to master state. The MESHLIB_THIRDPARTY_SKIP_BUILD short-circuit is no longer needed -- the workflow now gates the build step directly. - Split the old "Install thirdparty libs" step into two: 1. "Install brew requirements" -- always runs (cached .dylibs link against brew libs at runtime, so packages must be on hosted runners even on cache hit). Calls install_brew_requirements.sh directly. 2. "Build thirdparty libs" -- gated with `if: steps.thirdparty-cache.outputs.cache-hit != 'true'`. Calls build_thirdparty.sh, which still invokes install_brew_requirements.sh internally (one redundant invocation on the rare cache-miss path). - Merge "Compute brew-prefix hash" + "Compute thirdparty submodule SHA" into one step "Compute cache key inputs" (id: cache-keys) with two outputs (brew-hash, thirdparty-src-hash). Build MRBind's reference updated from `steps.brew-hash.outputs.hash` to `steps.cache-keys.outputs.brew-hash`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

oitel · 2026-05-15T08:51:31Z

+          echo "brew-hash=$(printf %s "$BREW_PREFIX" | shasum -a 256 | cut -c1-16)" >> "$GITHUB_OUTPUT"
+          echo "thirdparty-src-hash=$(git ls-tree HEAD \


Is it safe to compute a unified hash based on these two?

oitel · 2026-05-15T09:23:15Z

+          path: |
+            lib
+            include
+          key: v1-thirdparty-${{ matrix.instance }}-${{ matrix.compiler }}-${{ steps.cache-keys.outputs.brew-hash }}-${{ steps.cache-keys.outputs.thirdparty-src-hash }}-${{ hashFiles('scripts/build_thirdparty.sh', 'scripts/install_brew_requirements.sh', 'scripts/thirdparty/clip.sh', 'scripts/thirdparty/fastmcpp.sh', 'thirdparty/CMakeLists.txt', 'requirements/macos.txt') }}


Can the hashFiles(...) be computed in the cache-keys step too?

Address review feedback on the cache-keys step: instead of two separate outputs feeding into the cache key alongside an inline hashFiles(...), compute a single `thirdparty-hash` output that incorporates everything that should bust the thirdparty cache when changed: - brew prefix (baked into produced .dylibs' rpaths) - submodule SHAs pinned in HEAD (the source we compile) - scripts/build_thirdparty.sh, scripts/install_brew_requirements.sh, scripts/thirdparty/{clip,fastmcpp}.sh, thirdparty/CMakeLists.txt, requirements/macos.txt Cache key shortens to v1-thirdparty-${matrix.instance}-${matrix.compiler}-${thirdparty-hash} `brew-hash` is kept as a separate output since Build MRBind below consumes it on its own. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Fedr added disable-build-windows disable-build-linux-vcpkg disable-build-ubuntu-x64 disable-build-ubuntu-arm64 disable-build-emscripten labels May 14, 2026

Fedr and others added 2 commits May 14, 2026 21:04

ci: empty commit to trigger cache-hit run

43606a2

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

ci(macos): bump thirdparty cache to actions/cache@v5

7a5a727

Match the version already used by build-mrbind, install-msys2-mrbind, and install-cuda in this repo. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

oitel reviewed May 15, 2026

View reviewed changes

oitel approved these changes May 15, 2026

View reviewed changes

Fedr merged commit 31b5818 into master May 15, 2026
25 checks passed

Fedr deleted the ci/cache-macos-thirdparty branch May 15, 2026 10:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci(macos): cache thirdparty lib/include outputs across runs#6100

ci(macos): cache thirdparty lib/include outputs across runs#6100
Fedr merged 5 commits into
masterfrom
ci/cache-macos-thirdparty

Fedr commented May 14, 2026 •

edited

Loading

Uh oh!

oitel May 15, 2026

Uh oh!

oitel May 15, 2026

Uh oh!

oitel May 15, 2026

Uh oh!

oitel May 15, 2026

Uh oh!

oitel May 15, 2026

Uh oh!

oitel May 15, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		echo "brew-hash=$(printf %s "$BREW_PREFIX" \| shasum -a 256 \| cut -c1-16)" >> "$GITHUB_OUTPUT"
		echo "thirdparty-src-hash=$(git ls-tree HEAD \

Conversation

Fedr commented May 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What changed in the workflow

Measured impact

What gets cached

Cache key

Test plan

Uh oh!

oitel May 15, 2026

Choose a reason for hiding this comment

Uh oh!

oitel May 15, 2026

Choose a reason for hiding this comment

Uh oh!

oitel May 15, 2026

Choose a reason for hiding this comment

Uh oh!

oitel May 15, 2026

Choose a reason for hiding this comment

Uh oh!

oitel May 15, 2026

Choose a reason for hiding this comment

Uh oh!

oitel May 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Fedr commented May 14, 2026 •

edited

Loading