tools: Add adaptive Bazel wrapper#1905
Draft
jrajahalme wants to merge 1 commit into
Draft
Conversation
jrajahalme
force-pushed
the
bazel-adaptive-wrapper
branch
from
057c7d1 to
89f4d3f
Compare
tools/bazel_adaptive.py runs Bazel with a concrete integer --jobs value, monitors progress output and memory pressure, stops and resumes bazel build processes as needed under memory pressure, and if that does not relieve memory pressure, restarts with lower or higher parallelism as memory allows. Integer --jobs values are used directly; Bazel-style HOST_CPUS and HOST_RAM values with optional multipliers, such as HOST_CPUS*.5 and HOST_RAM*.0002, are resolved once as the maximum adaptive jobs cap. The wrapper rewrites bazel output to indicate the number of paused jobs (if any). "(13 actions, 12 running)" is shown as "(13 actions, 2 paused, 10 running)" when 2 of the running actions have been paused by the wrapper. This wrapper is only modestly faster than builds with carefully tuned --jobs count and ample available swap space. The main benefit of this wrapper is that it removes the tuning work and automatically restarts bazel when if fails for out-of-memory or job termination. To allow Bazel to get through memory bottlenecks during the build without unnecessary restarts the build environment should have swap available. This wrapper was tested on a 24 GB VM with 16 GB swap. Running the wrapper with BAZEL_ADAPTIVE_DISABLE_PTY=1 defined allows more of the diagnostic lines to remain visible even when running in an interactive terminal. Each diagnostic line starts with "[bazel-adaptive/<XXX>s]", where "<XXX>" is the number of seconds since the wrapper started. This commit changes the Makefile to use the wrapper by default. Written by Codex through an iterative session where codex has run the included Python tests and has observed the behavior of live runs to observe the runtime behavior. Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>
jrajahalme
force-pushed
the
bazel-adaptive-wrapper
branch
from
89f4d3f to
68e2640
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
tools/bazel_adaptive.py runs Bazel with a concrete integer --jobs value, monitors progress output and memory pressure, stops and resumes bazel build processes as needed under memory pressure, and if that does not relieve memory pressure, restarts with lower or higher parallelism as memory allows. Integer --jobs values are used directly; Bazel-style HOST_CPUS and HOST_RAM values with optional multipliers, such as HOST_CPUS*.5 and HOST_RAM*.0002, are resolved once as the maximum adaptive jobs cap.
The wrapper rewrites bazel output to indicate the number of paused jobs (if any). "(13 actions, 12 running)" is shown as "(13 actions, 2 paused, 10 running)" when 2 of the running actions have been paused by the wrapper.
This wrapper is only modestly faster than builds with carefully tuned --jobs count and ample available swap space. The main benefit of this wrapper is that it removes the tuning work and automatically restarts bazel when if fails for out-of-memory or job termination.
To allow Bazel to get through memory bottlenecks during the build without unnecessary restarts the build environment should have swap available. This wrapper was tested on a 24 GB VM with 16 GB swap.
Running the wrapper with BAZEL_ADAPTIVE_DISABLE_PTY=1 defined allows more of the diagnostic lines to remain visible even when running in an interactive terminal. Each diagnostic line starts with "[bazel-adaptive/s]", where "" is the number of seconds since the wrapper started.
This commit changes the Makefile to use the wrapper by default.
Written by Codex through an iterative session where codex has run the included Python tests and has observed the behavior of live runs for runtime behavior.