Skip to content

Reduce stage log Put allocation overhead#7027

Open
rootp1 wants to merge 1 commit into
pipe-cd:masterfrom
rootp1:fix/stagelog-put-alloc-overhead
Open

Reduce stage log Put allocation overhead#7027
rootp1 wants to merge 1 commit into
pipe-cd:masterfrom
rootp1:fix/stagelog-put-alloc-overhead

Conversation

@rootp1

@rootp1 rootp1 commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Summary

  • reduce stage-log Put overhead by encoding log blocks directly into the final buffer instead of allocating a []byte per block
  • add a filestore compatibility test that locks the persisted line-delimited format and EOL marker
  • add a benchmark that compares the legacy and current write paths for larger stage-log fragments

Linked Issue

Fixes #7026

What Changed

  • replace per-block json.Marshal plus buf.Write with a single json.Encoder writing into the destination buffer
  • add TestFileStorePut to verify persisted output remains backward compatible
  • add BenchmarkStageLogFileStorePut to capture the allocation and latency improvement against the legacy implementation

Verification

  • go test ./pkg/app/server/stagelogstore — passed
  • go test -run '^$' -bench BenchmarkStageLogFileStorePut -benchmem ./pkg/app/server/stagelogstore — passed
  • make build/go — passed
  • make check/gen/code — passed
  • make check — failed: yarn is not installed in this environment, so build/web could not run
  • make lint/go — failed: the pinned golangci-lint image is built with Go 1.25, but the repository targets Go 1.26.2
  • make test/go — failed: unrelated pre-existing repository test failures in pkg/app/piped/executor/kubernetes and Darwin tool tests in pkg/app/piped/platformprovider/kubernetes
  • make check/dco — failed: upstream history on this branch contains commit 25ae69937e55e8330bff96c34346718f3e5e7148 without Signed-off-by, and rewriting shared upstream history is not safe

Scope

  • only pkg/app/server/stagelogstore implementation and tests were changed; unrelated files were not modified

What this PR does:

Reduces allocation overhead in stage-log filestore persistence and adds regression coverage for the persisted format and benchmarked write path.

Why we need it:

The existing Put path rebuilt the full payload with per-block json.Marshal allocations on every write. This keeps the format the same while lowering per-write allocation cost.

Which issue(s) this PR fixes:

Fixes #7026

Does this PR introduce a user-facing change?:

  • How are users affected by this change: Stage-log persistence is more efficient under larger or frequently updated logs
  • Is this breaking change: no
  • How to migrate (if breaking change): not applicable

Signed-off-by: rootp1 <arnav.iitr@gmail.com>
@rootp1
rootp1 requested a review from a team as a code owner July 13, 2026 08:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reduce stage log Put allocation overhead in filestore persistence

1 participant