Reduce unnecessary CI runtime#2151
Open
pranaygp wants to merge 4 commits into
Open
Conversation
🦋 Changeset detectedLatest commit: 681afae The changes in this PR will be included in the next version bump. This PR includes changesets to release 0 packagesWhen changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
Contributor
🧪 E2E Test Results❌ Some tests failed Summary
❌ Failed Tests▲ Vercel Production (5 failed)fastify (1 failed):
hono (1 failed):
nextjs-webpack (1 failed):
nitro (2 failed):
Details by Category❌ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 📋 Other
❌ Some E2E test jobs failed:
Check the workflow run for details. |
Contributor
📊 Benchmark Results
workflow with no steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express workflow with 1 step💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express workflow with 10 sequential steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express workflow with 25 sequential steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express workflow with 50 sequential steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express Promise.all with 10 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express Promise.all with 25 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express Promise.all with 50 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express Promise.race with 10 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express Promise.race with 25 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express Promise.race with 50 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express workflow with 10 sequential data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express workflow with 25 sequential data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express workflow with 50 sequential data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express workflow with 10 concurrent data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express workflow with 25 concurrent data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express workflow with 50 concurrent data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express Stream Benchmarks (includes TTFB metrics)workflow with stream💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express stream pipeline with 5 transform steps (1MB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express 10 parallel streams (1MB each)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express fan-out fan-in 10 streams (1MB each)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express SummaryFastest Framework by WorldWinner determined by most benchmark wins
Fastest World by FrameworkWinner determined by most benchmark wins
Column Definitions
Worlds:
❌ Some benchmark jobs failed:
Check the workflow run for details. |
pranaygp
force-pushed
the
pranaygp/codex/ci-runtime-fast-paths
branch
from
d162d1d to
71b6435
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This PR reduces GitHub Actions CI runtime by introducing a “fast-path” for docs/UI-only pull requests, gating expensive benchmark lanes behind explicit opt-in, and deduplicating repeated E2E package builds by sharing build artifacts across E2E jobs.
Changes:
- Add a
ci-scopejob to classify PR changes and skip most CI/benchmarks when only docs/UI files changed. - Require manual dispatch or an explicit
community-benchmarkslabel before running community-world benchmark jobs. - Build E2E package artifacts once (Linux) and reuse them across local E2E lanes; canary lanes still rebuild.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 6 comments.
| File | Description |
|---|---|
packages/utils/src/get-port.test.ts |
Relaxes a Windows timing assertion to reduce CI flakiness. |
.github/workflows/tests.yml |
Adds CI scoping + shared E2E package build artifacts and updates job gating/required-check behavior accordingly. |
.github/workflows/benchmarks.yml |
Adds CI scoping + opt-in gating for community-world benchmarks and skips ordinary benchmarks on fast-path PRs. |
.changeset/ci-runtime-fast-paths.md |
Records the CI behavior change via a changeset entry. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Contributor
karthikscale3
left a comment
karthikscale3
left a comment
There was a problem hiding this comment.
ai review: regression/coverage pass on the CI-runtime changes. The artifact sharing and benchmark opt-in look sound; main concern is the fast-path classification removing all test/type coverage for packages/web*. Notes inline (a couple marked optional).
TooTallNate
reviewed
May 29, 2026
Member
TooTallNate
left a comment
TooTallNate
left a comment
There was a problem hiding this comment.
Comment-only — directionally good, one valid concern from @karthikscale3 worth resolving
The CI bill numbers in the PR body are eye-opening (~6.2k runner-minutes on community benchmarks that never finish, ~3.8k on duplicated package builds). The three changes are well-targeted:
- Community benchmarks opt-in via
community-benchmarkslabel or manual dispatch — correct, given the 60-minute timeouts with zero success in the 24h sample - Fast-path classification for docs/UI-only PRs
- Shared E2E package build artifact with stable lanes downloading once and canary lanes rebuilding
Verified locally / against CI
pnpm --filter @workflow/utils test✓ (94/94 — the Windows timeout bump from 2s → 5s holds)- CI: 108 success, 0 failures, 8 skipped on the latest run, with all 65 E2E lanes (Local Dev/Prod/Postgres + Vercel + Windows) + Unit + Build Shared E2E all green. The shared-artifact extraction empirically works.
- Branch protection: I checked the active rulesets — only
DCOandE2E Required Checkare required status checks onmain. So karthik's branch-protection concern doesn't apply to this repo's config (the meta-check correctly fast-paths and that's the only required status check). Worth keeping that invariant in mind if branch protection ever changes to require individual lanes. - Copilot's earlier artifact-nesting concern was empirically resolved by
b81601b35 Fix shared E2E artifact extraction path— the green E2E lanes confirm extraction works correctly.
Karthik's "unit job skipped for web/web-shared changes" concern is valid
@karthikscale3 flagged that packages/web/ and packages/web-shared/ in the fast-path patterns means the unit job (which runs pnpm test --filter='!./docs' --filter='!./workbench/vitest') is skipped for those PRs. But @workflow/web-shared has unit tests that the unit job covers — for example, the trace viewer tests, event-list duration tests, etc. With the fast-path, those tests silently don't run for any PR that only touches web-shared.
Two options:
-
Remove
packages/web/andpackages/web-shared/from the fast-path patterns so the unit job continues to run for those changes. Loses some perf for UI PRs but keeps coverage. This is probably the right call — the original perf win is dominated by E2E lanes, not unit. -
Keep them but bump web-shared tests into the fast-path category somehow (e.g. a smaller
web-unitjob that only runs the web/web-shared package tests). More plumbing, more value preserved.
I'd lean (1) — the unit job is cheap relative to E2E, and silently skipping it for any PR is a real coverage gap.
Karthik's "hand-maintained allowlist drifting from turbo.json" — agreed, low priority
The artifact upload paths (packages/*/dist, packages/*/.well-known, packages/*/src/version.ts, swc-plugin-workflow.wasm, etc.) duplicate turbo.json's build.outputs. If a future package adds a new output dir (e.g. lib/), the artifact won't include it and the lane will silently rebuild. Worth a comment pointing at turbo.json as the source of truth, but probably not worth dynamically deriving the patterns now.
Karthik's "duplicated ci-scope across tests.yml and benchmarks.yml" — agreed, optional
The ci-scope job is byte-identical in both workflow files. If the fastPathPatterns list drifts between them, behavior diverges. Could be extracted to a reusable composite action (.github/actions/detect-ci-scope/). Optional cleanup.
Commenting (not approving) until the web-shared unit coverage gap is resolved
Once you either drop packages/web/ and packages/web-shared/ from the fast-path patterns OR add a targeted unit job that runs for them, I'm happy to approve. The other concerns are non-blocking.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
community-benchmarkslabel before community-world benchmark jobs runWhy
I inspected the last 24 hours of Actions data before making these changes. Community-world benchmark jobs accounted for roughly 6,174 runner-minutes while repeatedly reaching their 60-minute timeout with no successful Redis, MongoDB, or Redis + BullMQ jobs in the sampled period. Local E2E lanes also repeated the same initial package build, totaling roughly 3,839 runner-minutes in ended runs.
This PR intentionally leaves a Vitest upgrade and smaller sleep/timing cleanups out of scope so the workflow behavior and resulting time reduction are easier to validate.
Validation
.github/workflows/tests.ymland.github/workflows/benchmarks.ymlwith Ruby YAML parsinggit diff --checkactionlintagainst the edited workflows and compared againstorigin-https/main; remaining diagnostics are pre-existing baseline diagnosticspnpm turbo run build --filter='!./workbench/*' --output-logs=errors-onlysuccessfully (29 scheduled tasks succeeded in about 59 seconds)pnpm changeset status --since=origin-https/mainfor the CI-only empty changesetmainruleset requiresDCOandE2E Required Checkonly; individual skipped E2E lanes are not required checksCI observation
Monitoring confirmed that stable Linux local E2E lanes restore
e2e-package-build-artifactsintopackages, Next.js canary lanes continue to rebuild, Vercel-prod lanes start independently of the shared build job, and community-world benchmarks are skipped without the opt-in label.E2E Required Checkpassed after hosted E2E retries. The Vercel Express and Nitro benchmark lanes continue to expose a pre-existing hosted stream-truncation failure outside this workflow diff.