docs(rfc): split TransformConfig build lifecycle into four phases#25848
Draft
pront wants to merge 5 commits into
Draft
docs(rfc): split TransformConfig build lifecycle into four phases#25848pront wants to merge 5 commits into
pront wants to merge 5 commits into
Conversation
Replaces the two-step actions/cache save+restore for ~/.cargo with
Swatinem/rust-cache, which additionally caches target/ build artifacts.
Design:
- Cache key = prefix + rustc hash + Cargo.lock hash (Swatinem default).
PRs with unchanged Cargo.lock get an exact hit; PRs that bump
Cargo.lock fall back to Swatinem's restore-keys prefix match and
reuse most of master's target/.
- save-if: master only. PRs restore but don't write.
- cache-bin: false. Setup action installs cargo bins explicitly; letting
Swatinem manage ~/.cargo/bin risks removing tools on persistent
self-hosted release builders.
Validated on vectordotdev/ci-sandbox with a minimal binary using the
same rdkafka feature set:
cold 120s
PR, src-only change 12s (exact key hit, -90%)
PR, Cargo.lock bump 9s (restore-key prefix hit, -93%)
Addresses three findings from the PR review: 1. Save on merge-queue runs (not only master). test.yml runs on pull_request + merge_group and never on master pushes, so cache-producing jobs like check-clippy never had a producer under the previous save-if: master-only condition. Merge-queue commits are the ones fast-forwarded onto master, so their caches are the closest possible representation of master state. 2. Trigger test workflow on setup action edits. The setup composite action's path was absent from changes.yml, letting setup changes ship without exercising the heavy consumers (clippy, tests). Added .github/actions/setup/** to the test-yml filter. 3. Differentiate matrix cache keys. Introduces optional cache-key-suffix input, passed through as Swatinem's 'key'. build-linux-cross's six targets share the same GITHUB_JOB and previously collided on the same cache key; now each target owns its own entry.
Switches Swatinem/rust-cache to shared-key 'workspace' so all Rust jobs restore from one common cache entry. Removes the merge_group save trigger: GitHub scopes caches by ref, and merge-queue refs are isolated from master and PRs, so saves there aren't visible to any consumer. Producer for this cache is the new warm-cache.yml workflow (separate commit), which runs on master and produces .rlib artifacts consumers can link against.
Producer workflow for the shared cache introduced in the previous
commit. Runs on:
- daily schedule (0 5 * * *)
- push to master when Cargo.lock or toolchain files change (for
immediate refresh after dep bumps)
- manual workflow_dispatch
Seed command is 'cargo build --workspace --tests --locked' so cached
artifacts include .rlib files (not just .rmeta), which is required
for consumers like 'cargo test' to link deps without recompiling
them. Validated on ci-sandbox with a kafka-demo binary using the same
rdkafka feature set as Vector's kafka source/sink:
Cold cargo build --tests 68s
PR cargo test (warm, .rlib cache) 8s (only workspace crate recompiled)
Compared with the wrong seed verb (cargo check, produces only .rmeta):
PR cargo test (warm, .rmeta only) 50s (all deps recompiled — no reuse)
github-actions
Bot
added
domain: ci
|
pront
added
the
no-changelog
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an RFC proposing to split the monolithic
TransformConfig::build()into four explicit lifecycle phases:validate_structure,validate_environment,build(pure/sync), andstart.Motivation:
build()currently conflates structural validation, environment validation, pure construction, and task spawning. This causes:vector validateunable to exercise VRL compilation without a workaround (validate_env(), context key guards, stub enrichment tables)Scope:
TransformConfigtrait and all existing transforms.SourceConfig/SinkConfigare deferred.Vector configuration
N/A -- no behavior change.
How did you test this PR?
RFC only -- no code changes.
Change Type
Is this a breaking change?
Does this PR include user facing changes?
no-changeloglabel to this PR.