Skip to content

Use parallel parsing at all stages#21266

Open
ilevkivskyi wants to merge 3 commits intopython:masterfrom
ilevkivskyi:parse-parallel-in-workers
Open

Use parallel parsing at all stages#21266
ilevkivskyi wants to merge 3 commits intopython:masterfrom
ilevkivskyi:parse-parallel-in-workers

Conversation

@ilevkivskyi
Copy link
Copy Markdown
Member

@ilevkivskyi ilevkivskyi commented Apr 18, 2026

This is a follow-up to #21175 and #21119.

Right now we are only using parallel parsing during initial graph loading. This PR allows using it when processing SCCs that are stale due to dependencies (these are not parsed on initial loading). Implementation is a bit less trivial than I thought, because we need to completely extract the tree de-serialization (which is not parallelizeable). It is still relatively straightforward. The new logic is like this:

  • When calling parse(eager=False) (default) we always return a serialized tree.
  • When calling parse(eager=True) a caller can force immediate de-serialization (when suitable), this will trigger load_from_raw().
  • To get more manual control (e.g. for imports_only=True) one can use parse(eager=False) followed by load_from_raw() directly.

cc @JukkaL

@github-actions
Copy link
Copy Markdown
Contributor

According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant