perf(docker): share reactor build across images - #3171
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #3171 +/- ##
============================================
+ Coverage 32.66% 37.76% +5.09%
- Complexity 5500 6539 +1039
============================================
Files 789 800 +11
Lines 67703 68882 +1179
Branches 8945 9147 +202
============================================
+ Hits 22116 26010 +3894
+ Misses 42983 39821 -3162
- Partials 2604 3051 +447 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
085106c to
8954c99
Compare
8954c99 to
c522d73
Compare
bitflicker64
left a comment
There was a problem hiding this comment.
Independently re-verified the core claims. The build stage is byte-identical across all four Dockerfiles at c522d73 (no pre-FROM ARGs, single root .dockerignore), SOURCE_REVISION reaches every Bake target and expands in the cache mount id at runtime, and only the build-cache target writes the shared cache. The timing table matches raw job-step timestamps from the linked hugegraph/actions runs: 1754s -> 330s (dry-run) / 501s (first publish) on the same runner class, with the Maven stage fully rebuilt in both candidate runs. Those runs built 89a2586, whose Dockerfiles and docker-bake.hcl are byte-identical to this head; the strengthened CI guard is covered by the Docker Build CI run on the head itself. Ran buildx bake --print locally: the jq assertion fails on platform or output mutations, and the awk stage-diff catches a divergent mvn line. One hardening nit inline; non-blocking.
Two observations, no action needed in this PR:
- The per-module
buildcache-*refs are read but no longer refreshed on the Bake path, so runtime-stage layers will slowly go stale and eventually rebuild each run. If that ever shows in timings, runtime targets could export a cache gated onEXPORT_CACHE. - Local builds share
id=hugegraph-maven-localacross revisions, andmvn installwrites SNAPSHOTs into that shared.m2. A full-reactor build resolves modules from the reactor, so the practical risk is low; worth a doc line at most.
imbajin
left a comment
There was a problem hiding this comment.
Blocking: yes. Summary: The default multi-platform Bake graph configures every runtime target to use the single-platform Docker exporter, so it cannot produce the claimed amd64/arm64 images. Evidence: docker buildx bake --print at c522d73 expands the runtime targets with platforms=[linux/amd64,linux/arm64] and output type=docker; the Docker exporter supports single-platform results.
# Conflicts: # .github/workflows/docker-build-ci.yml # hugegraph-server/Dockerfile-hstore
- reject an empty shared-stage extraction - document the containerd load-before-push path - keep the evaluated Bake graph unchanged
- export runtime layers to per-image registry caches - keep shared Maven cache export isolated - avoid broad Dockerfile CI path matching
- use ubuntu-latest for Docker CI jobs - keep Docker README prose on single lines - preserve code blocks, tables, and document structure
- move the Bake definition to docker/bake.hcl - update Docker CI to use the explicit file path - document root-level inspection and build commands
- move build and contribution guides under docs - update repository links to the new paths - remove tracked IntelliJ metadata and ignore .idea
- link the merged Bake configuration path - document host-platform builds for local changes - clarify cache and publication behavior
- move routine PD watch events from INFO to DEBUG - move per-task scheduler lifecycle logs from INFO to DEBUG - keep warnings, failures, and lifecycle signals unchanged
- preserve production task lifecycle messages at INFO - suppress scheduler INFO only during the HStore core test - stop PD and Store startup scripts from echoing their source
Summary
docker/bake.hclso BuildKit can solve the PD, Store, HStore Server, and standalone Server images togetherCompanion workflow PR: hugegraph/actions#27
Before → after
The portable Java build runs once on
BUILDPLATFORM. Only the runtime stages fan out for amd64 and arm64, so the expensive Maven work stays native instead of running again through QEMU.Benchmark
Both measurements used cold caches, the same runner class, the same four images, the same functional checks, and a real Docker Hub publish. Queue time is excluded.
Design
BUILDPLATFORMkeeps Maven native; runtime stages use the correct target-platform JRE and only the ARM runtime work uses QEMUmode=max; each runtime image writes its ownmode=mincacheCI checks that the shared stages remain identical and validates the evaluated Bake targets, platforms, outputs, and cache layout. The published test images contain both linux/amd64 and linux/arm64 variants. Functional Compose, Gremlin, and standalone checks run against the local amd64 variants before any image is pushed.
Repository housekeeping in this PR moves the build and contribution guides under
docs/, updates their references, removes the tracked IntelliJ project metadata, and limits HStore test noise without changing production task lifecycle logging.Only temporary test tags were published.
latestwas not changed.