perf: walk the file system off the thread webpack builds on - #357
Merged
Merged
Conversation
`collectFromFileSystem` walked the whole tree with `globSync` from inside the `compilation` hook, so every compilation stopped the loop for as long as the walk took — 6.0 ms over a thousand files, and the walk is made again on every rebuild. `canWatch` read each candidate directory with `readdirSync`, and `parseFoldersToGlobs` stated each pattern, both from places that can wait. All three are asynchronous now. Two compilers asking for the same walk at the same moment share it rather than reading the tree twice, which is also what leaves their checks a run to share. Over a thousand files the longest stall in a watch rebuild falls from a median of 10.2 ms to 5.8 ms, measured over nine rebuilds twice. What stays synchronous is TypeScript's: its host is called for a source file and waits for one, so `versionOf` and `ts.sys` have nothing to await into — and since the check moved to a worker that is not this loop.
This was referenced Sep 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
collectFromFileSystemwalked the whole tree withglobSyncfrom inside thecompilationhook, so every compilation stopped the loop for as long as the walk took — and the walk is made again on every rebuild.canWatchread each candidate directory withreaddirSync, andparseFoldersToGlobsstated each pattern; both sit in places that can wait. All three are asynchronous now.Measured over a generated 1000-file tree:
globSyncglobEnd to end, the longest event-loop stall during a watch rebuild, nine rebuilds per run, two runs of each arm:
A full build shows no difference — webpack's own work is the longest stall there — so the win is in rebuilds, which is where the walk is repeated and webpack has little to do.
Two compilers asking for the same walk at the same moment now share it rather than reading the tree twice. That also keeps
multi-compiler's run sharing working: the walks used to finish in lockstep because they were synchronous, and without sharing the second compiler arrived after the first check's run had already settled — the existing test caught it.For scale, the two small ones:
readdirSyncis 0.011 ms a call andstatSync0.002 ms, so they are converted because they cost nothing to convert, not because they showed up.What stays synchronous is TypeScript's. Its host is called for a source file and must return one, so
versionOfand the eightts.syscalls have nothing to await into — and since #343 that work is on a worker, so it is not this loop. There is a comment saying so now.What kind of change does this PR introduce?
perf.
Did you add tests for your changes?
No new ones — this changes no behaviour, and the existing suite covers it, including the
multi-compilercase that failed on the first attempt.test/utils.test.jsis updated for the now-asyncparseFoldersToGlobs.Does this PR introduce a breaking change?
parseFoldersToGlobsis async now; it is not exported from the package entry.If relevant, what needs to be documented once your changes are merged or what have you already documented?
n/a.
Use of AI
AI was used. It was asked to take the synchronous calls off the build's thread, measured each one first to decide which were worth converting, made the change, and measured the rebuild stall before and after in two paired runs.
🤖 Generated with Claude Code
https://claude.ai/code/session_01GzZci4NQeiqwdrVfd7dGXy
Generated by Claude Code