Skip to content

Reduce matrix scheduling time and preserve queued work - #801

Open
thomashoneyman wants to merge 2 commits into
masterfrom
fix/matrix-scheduling
Open

thomashoneyman wants to merge 2 commits into
masterfrom
fix/matrix-scheduling

Conversation

@thomashoneyman

@thomashoneyman thomashoneyman commented Oct 10, 2026 •

Copy link
Copy Markdown
Member

Partially addresses #800. In that issue, web-storage@5.0.1 finished publishing to the registry and Pursuit, then timed out while solving its follow-up matrix jobs. Since we solve all the followups before we insert any jobs this means we ended up with zero matrix jobs queued. This doesn't affect the package publication per se, but it means the supported compilers list ends up being wrong. Retrying publication doesn't help because it just returns that the package was already published.

There are two problems at play. First, calculating the matrix cascade is very slow. Second, if there is an issue and we don't schedule all the jobs, there's no way to do it after the fact. This change addresses the first problem and preserves work already queued. Here's what we do:

  1. We speed things up. We read one registry snapshot per scheduling pass and sort the manifest index once per dependent cascade rather than sorting it for each recursive lookup.
  2. We insert matrix jobs as their build plans get solved so if there is an interruption, the already-queued work isn't affected.

On the current registry index, using this method for web-storage@5.0.1 drops the cascade from 102.5 seconds before this change to 5.66 seconds afterward, about 18 times faster. Both runs produced the same 14 complete build plans, including dependency resolutions. The first plan was emitted after 0.37 seconds.

Note: this doesn't add any kind of background scheduler. If work didn't get queued before an interruption it still won't be recovered automatically. Fixing that is a bigger job and probably needs a lil discussion.

@thomashoneyman thomashoneyman changed the title Reduce matrix scheduling time and recover publish retries Reduce matrix scheduling time and preserve queued work Oct 11, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant