feat(bundle): bundle the bit cli with esbuild - #10590
Draft
GiladShoham wants to merge 342 commits into
Draft
Conversation
…e links when hoisted install fails
…ons only for legacy core envs
…eambit/bit into remove-core-envs-from-manifest
…hes, guard preview scaling for legacy envs
…env-chain installs
…om-manifest # Conflicts: # .circleci/config.yml
- isLegacyCoreEnv and related helpers strip the version internally (like isCoreEnv) - drop dead legacy-slot fallbacks already covered by getEnvDefinitionByStringId - share the in-flight load key between workspace/scope aspect loaders - reuse isAspectLoadedIgnoringVersion instead of an inline harmony scan - extract calcMergedRootPolicy, de-async a sync loop, memoize the fallback compiler - use Sets in GraphFromFsBuilder membership checks, warn once on multi-version envs
Also fix node-modules-linker to never persist a source-file main (install's early linking step runs before component load, so package.json main was written as index.ts, breaking every subsequent component require)
# Conflicts: # scopes/ui-foundation/ui/bundle-ui.task.ts # scopes/ui-foundation/ui/ui.main.runtime.ts
Superseded by #10609, which replaced uidNumber() with an in-process /etc/group read, so the child-process require the externals entry existed for no longer happens.
…nent The helper now lives in its own component, published as @teambit/webpack.modules.exclude-node-modules-js. Both bundler configs import it from there and the copy that lived in the webpack aspect is gone, along with its re-export - nothing outside those two configs used it. Pinned at 0.0.1 in the workspace policy. The lockfile is deliberately untouched: regenerating it here rewrites ~130k lines of unrelated drift, which belongs in its own chore commit.
Adds @teambit/webpack.modules.exclude-node-modules-js@0.0.1, pinned in the previous commit's policy entry. The diff is far larger than that one package: this is the first `bit install` to reconcile the lockfile with the branch's `workspace.jsonc`, so it also drops entries the branch had already removed - `@teambit/aspect` from the importers, and the `@teambit/legacy@2.1.0` peer suffixes that trail through most resolutions.
Review: BundleUiTask found the first rejected root, threw its reason, and the catch
immediately replaced it with `new Error("Generating UI bundle failed")`. The actual
error only ever reached the debug log - diagnosing a failed BundleUI in CI meant
downloading the log artifact to find out what rspack actually said.
Now the thrown message names the roots that failed, every failing root is logged
(not only the one that ends the task), and the first reason is attached as `cause`.
The reason itself stays out of the message on purpose: for an rspack failure it is
the entire stats output, megabytes of it, which is why the message was generic in
the first place.
Same issue as BundleUI, found by review: `buildPreBundlePreview` creates an rspack compiler, runs it, and never closes it - so the module graph and rspack native side stay resident. Both callers keep the process alive well past that point: the PreBundlePreview build task runs once per env inside a `bit build` that runs every task in one process, and `bit start` pre-bundles on demand in a server that then keeps running. Closed in a `finally`, so the error paths release it too, and the close itself is guarded - cleanup must not replace the build error that sent us there. No deferral needed here, unlike BundleUiTask: there is one compiler and nothing else bundling alongside it. Checked the other rspack call site while here: `ui-server.ts` hands its compiler to RspackDevServer, which needs it alive to watch, so it stays as is.
Conflict resolutions: - .circleci/config.yml: master's revert of --keep-lane (#10615) applied inside this branch's memory-sampler wrapper for the `bit ci pr` step. - scopes/react/react/react.env.ts, react.main.runtime.ts: stay deleted - this branch removes the core env sources from the workspace, so #10610's edits to them are moot. - scopes/preview/preview/rspack/rspack.config.ts: keep the mdxOptions parameter. The import master re-adds is what this branch removed on purpose: @teambit/mdx.modules.mdx-v3-options is ESM-only and has to reach the config through a native import() done by the caller. - scopes/preview/preview/pre-bundle.ts: keep master's closeRspackCompiler, which logs a close failure instead of swallowing it (#10612), and drop this branch's earlier copy. - workspace.jsonc: both sides - master's webpack-bundler/webpack-dev-server entries plus this branch's exclude-node-modules-js. - pnpm-lock.yaml: regenerated from master's lockfile with `bit install`, so the full-re-resolution lockfile this branch carried is gone. @teambit/react is now the only added package (1.0.1099, consumed from the registry); everything else is importers and duplicate versions dropping out with the core env sources. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…om-manifest # Conflicts: # .bitmap
@teambit/node, react, aspect, env: 1.0.1042 -> 1.0.1102 @teambit/mdx, readme: 1.0.1043 -> 1.0.1103
…om-manifest # Conflicts: # .bitmap # .circleci/config.yml # pnpm-lock.yaml # scopes/harmony/aspect/aspect.env.ts # scopes/react/react/apps/web/react-app-options.ts # scopes/react/react/apps/web/react.application.ts # scopes/react/react/apps/web/webpack/mutators.ts # scopes/react/react/react.env.ts # scopes/react/react/react.main.runtime.ts # scopes/react/react/webpack/webpack.config.env.dev.ts
@teambit/node, react, aspect, env: 1.0.1102 -> 1.0.1105 @teambit/mdx, readme: 1.0.1103 -> 1.0.1106 react/aspect no longer depend on @teambit/webpack after #10620.
… the core aspect WebpackMain.createBundler/createDevServer, the WebpackBundler/WebpackDevServer classes, and their config factories duplicated what @teambit/webpack.webpack-bundler and @teambit/webpack.webpack-dev-server now do (react/aspect/node envs build through those packages directly, not this aspect - see #10620). Nothing in the codebase calls createBundler/createDevServer anymore. Also drops generateExposePeersTransformer (already marked dead code) and its only helpers (get-exposed-rules, resolve-peer). Kept: the type exports, WebpackConfigMutator/Configuration re-exports, the thin transformer/plugin/fallback re-exports, and the WebpackMain class itself (gutted, so the aspect still loads for backward compatibility).
…apture check_circular_dependencies was the only bit-executing job left on the hub-stg (staging) default while bit_pr/bit_merge use production hub; building the workspace graph here can trigger a live scope import of uncached component objects, and staging appears to be the likely cause of the sporadic 10m no-output CI hangs. Also add a script-level timeout so a stall fails fast with a clear message, print the resolved bit binary/version/hub_domain for diagnostics, and persist debug.log as a CI artifact.
Mirrors e2e's --bit_bin. Lets us pin/compare which bit binary these scripts run against (repo build vs a bvm-linked release) instead of relying on whatever "bit" resolves to on PATH - useful for narrowing down the CI hang, which turned out to reproduce locally too: the repo's own binary took ~3x longer than a released bvm build on the same workspace (5m50s vs 1m57s for `insights circular --json`).
An auto-merge with bit-ci's version-bump commit reintroduced 7 top-level .bitmap entries (aspect, babel, env, mdx, node, react, readme) for legacy core envs this branch already removed from tracking - their source no longer exists at those paths, so bbit install failed with ComponentNotFoundInPath. The bot's commit predates this branch's removal work, so a naive 3-way merge kept its stale entries instead of the deletion. Removed exactly those 7 blocks; confirmed via diff against merge-base/mine/bot that nothing else in .bitmap needed touching (the remaining changes are the legitimate 2.0.80->2.0.81 version bump). bit status loads cleanly now.
…ce-cycle Verified against two real CI runs and a local repro: execSync's timeout error here doesn't set error.killed, it's a raw ETIMEDOUT from spawnSync, so the clearer message never fired. Also correct the message itself - the CI evidence points at a local force-write storm during aspect loading (every dist file of every loaded aspect written to every duplicate .pnpm-hash variant), not a network/registry stall.
Diagnostic addition to narrow down the perf regression found in this check: run it once with the repo's own binary (gating, as before) and once with the bvm-linked nightly release already fetched by setup_harmony (informational only, never fails the job). Reuses the existing install_bvm/bvm_upgrade commands and the same .bvm cache key setup_harmony populates this pipeline run, so it should mostly reuse the cached bundle rather than re-download. Remove once the regression in the repo binary's aspect-loading path is root-caused.
CircleCI skips steps after a failed one unless that specific step is
marked when:always - it doesn't propagate through a whole reusable
command's inner steps. install_bvm/bvm_upgrade sat after the repo-bit
check step, so when that step failed/timed out (as expected), bvm
setup never ran and the diagnostic comparison silently no-opped
("bbit: command not found", swallowed by the || true). Move the bvm
setup earlier so it runs unconditionally, independent of the repo-bit
check's outcome.
We already know the repo-bit check is slow/fails; running the known- good bvm baseline first gives that signal without waiting on the repo-bit run, and it stays non-blocking (|| true) so it can't skip the repo-bit gate either way.
Repo bit has a confirmed, reproducible perf regression on this check (~5min timeout vs ~1-2min on a bvm release, across 4 environments) - see the new CI-HANG-INVESTIGATION.md for the full root-cause writeup (narrowed to workspace.ts's self-as-aspect recursion, unlocked by isCoreAspect() now returning false for the envs this branch removed from the core manifest). Disabling the repo-bit step until it's fixed so it can't block merges on a broken check; the non-blocking bvm-bit comparison stays as a sanity signal. Re-enable once fixed.
Runtime-instrumented the workspace.ts self-as-aspect branch in a disposable /tmp clone - it's never entered during the hang, disproving the earlier hypothesis. Re-reading the captured debug.log's INFO-level loadAspects lines (not the much noisier DEBUG-level file-write lines) shows the real shape: one single trace root recurses for the entire 5-minute window, alternating consumer-fs-load/extension-merge calls into workspace.loadAspects without ever terminating - not fan-out across many components. Found a near-exact match already diagnosed and fixed (unmerged) on origin/refactor/component-loading-v2-take-3-stage2: commits f9ae003 and its follow-up 1213c36 describe and fix the identical mechanism in WorkspaceAspectsLoader.loadAspects (concurrent calls for different root aspects independently re-isolating a shared env dependency, because isAspectLoaded only flips true after a load completes). Their fix serializes loadAspects through a single queue; measured similarly (13:54 -> 10s on a 311-component workspace). Porting that approach is now the primary recommended next step.
…om-manifest # Conflicts: # .bitmap # pnpm-lock.yaml
@teambit/node, react, aspect, env: 1.0.1105 -> 1.0.1107 @teambit/mdx, readme: 1.0.1106 -> 1.0.1108
writePreviewEntry hashed the raw resolved-aspect set, which always includes workspace-specific aspects (e.g. teambit.react/react) the shipped pre-bundle was never built against, so the hash could never match and every build fell through to a full rebuild. Now also accepts a match against the same core-aspects-only hash PreBundlePreviewTask used to bake the artifact.
webpack-fallbacks-aliases.ts has two eager require.resolve()s at module scope, process/browser and buffer/, and blows up on the first missing one. bit tag --build against a webpack-based env's default bundler was hitting Cannot find module 'buffer/' right behind the already-externalized process/browser.
…externals Now that the react/node/aspect envs bundle component previews through the external @teambit/webpack.webpack-bundler package instead of WebpackMain.createBundler (removed upstream), webpack itself is never required at runtime, and the process/browser + buffer/ fallback paths that call site needed are unreachable from that flow too. Verified by rebuilding with all three removed and re-running the e2e test that originally caught the buffer/ gap.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bundles the CLI into a single 67 MB CJS file plus 11 externals that cannot be inlined, with a generated shim package per core aspect re-exporting its slice of the bundle.
npm run bundlebuilds it;--seaalso builds a node single executable.1.2 GB / 141k files → 231 MB / ~7.3k files.
bit --help0.70s → 0.64s.The shims emit the same
dist/*.aspect.jsanddist/*.main.runtime.jsfilenames the aspect loader already discovers, so the runtime needed no changes. The one source fix is real and independent of bundling:hook-requirepatchedmodule.constructor.prototype.require, which under any bundler installs an enumerablerequireonObject.prototype.Verified from an isolated dir: 40 commands including
create,status,tag,export,import,watch,server, andbuild --unmodified(all 9 tasks, rspack included). Not working:bit start— see the report.npm run e2e-test:bundle/:searun the suite against the artifact; CircleCI builds it once insetup_esbuild_bundleand shares it across the e2e nodes (gated to^bit-bundle.*branches).Full architecture, measurements, externals breakdown, script-vs-SEA analysis, the publishable package layout and open questions are in
bundle-plan.md.Draft: based on
remove-core-envs-from-manifest, so the diff includes that branch. Opened to get CircleCI running.