Throttle `node-test-pull-request` jobs on test CI

[sxa]

Executor count on both machines has been dropped back to 25.

I'm tempted to leave the parallelism restrictions on node-test-commit but I'll raise a separate issue to discuss at the next build WG meeting...

Originally posted by @sxa in #4432

I think it makes sense to keep the PR testing throttled. Some background and reasoning:

• We currently have 75 jenkins-workspace executors across three machines. The node-test-pull-request job spawns one node-test-commit on each run, and most of those will spawn off fanned jobs for windows and arm (*) and all of those use up one executor for each pull request testing job. The post-build-status-update job, when required, also requires a jenkins-workspace executor for a short period to do its work of posting back to the appropriate PR.
• Throttling node-test-pull-request to 10 would allow at most 50 of those 75 executors to be used for pull request testing, leaving 25 for other jobs that need it a jenkins-workspace node in the CI.
• I'll also note that while the platform subjobs such as node-test-commit-osx also do some work on the jenkins-workspace nodes they run via a lightweight executor that does not count towards the 25 on each node.
• Having it as more than 10 is probably counter-productive. While it would allow a higher throughput of some of the faster platforms, the ultimate reporting back of status to the PRs is only done when the jobs is complete, which is gated by the slowest platform. 10 is probably still quite generous given that we don't have more than two machines for any given label.
• The disk usage on the workspace servers increases if you allow more jobs to execute in parallel. If 10 pull requests are running in parallel the jenkins-workspace node (assuming the worst case that they all end up on the same node) will require ten separate working directories of up to 4GB each (+) - each will go into a directory suffixed by @n.
• As mentioned earlier the platform specific jobs use a lightweight executor on the jenkins-workspace nodes. They still require space and as an example the OSX job got up to node-test-commit-osx@33 today because so many were in the queue (it had previously reached 35 in September 2024). This demonstrates that without throttling at the top level you can chew up disk space on the workspace servers (each of those takes up 2.5-3GB)
• I will also note that as a side effect of throttling node-test-commit to 10 for the OSX problem we now have a backlog of over 10 in the queue. This is causing jobs not to be triggered purely due to the delay caused by the backlog of osx jobs. This number will increase over the day. We should consider this when deciding to throttle and perhaps make the metric of queued jobs which are throttled something to actively monitor.
• Reducing to 10 would likely decrease the probability of hitting the 20 minute timeout occurring on node-test-commit jobs (@)

As part of resolving the problem described in 4432 I throttled node-test-commit, but it makes more sense to throttle at the highest level - node-test-pull-request which implicitly limits those below it. This will resolve the possibility of deadlocks with the executors due to the need to spawn post-build-status-update at the end of the job and also

Side note: Moving to pipelines might help with this as they typically only use lightweight executors too, but that's a separate topic

(*) - I also note that arm32 fanned builds are triggered on node-daily-master even though we don't ship arm32 on that version - it looks like they're still passing though!)u
(+) - here is an example of the space usage from node-test-pull-request - as mentioned the osx one is worse at the moment as it goes up to @35 on the mnx-2 node:

iojs@test-mnx-ubuntu2204-x64-2:~/build/workspace$ du -sk node-test-pull-request* | grep -vw tmp
2333368 node-test-pull-request
2323504 node-test-pull-request@2
2426475 node-test-pull-request@3
2425107 node-test-pull-request@4
3959716 node-test-pull-request@5
3943765 node-test-pull-request@6
1538253 node-test-pull-request@7
1569737 node-test-pull-request@8

(@) - An example happened today in https://ci.nodejs.org/job/node-test-commit/90858/console:

11:45:46  > git fetch --no-tags --force --progress -- git@github.com:nodejs/node.git +refs/heads/*:refs/remotes/origin/* +refs/pull/65426/head:refs/remotes/origin/_jenkins_local_branch # timeout=20
12:05:46 ERROR: Timeout after 20 minutes
12:06:30 ERROR: Error cloning remote repo 'origin'
12:06:30 hudson.plugins.git.GitException: Command "git fetch --no-tags --force --progress -- git@github.com:nodejs/node.git +refs/heads/*:refs/remotes/origin/* +refs/pull/65426/head:refs/remotes/origin/_jenkins_local_branch" returned status code 128:

It had got close before it was killed - 90% through the final Resolving deltas phase:

Resolving deltas:  90% (774742/859022)
fatal: fetch-pack: invalid index-pack output

[sxa]

Noting from a slack conversation with @aduh95 and @targos that the code that posts to the originator's PR when request-ci is added is done by node-test-pull-request before node-test-commit is kicked off (Ref: https://github.com/nodejs/github-bot/blob/d04ac555a81e02d511eb983e6e72f024c342c217/lib/push-jenkins-update.js#L13-L17 - which is called from https://github.com/nodejs/build/blob/main/jenkins/pipelines/post-build-status-update.jenkinsfile - and the log snippet below):

15:17:37 [description-setter] Description set: Testing <a href=https://github.com/nodejs/node/pull/65416>65416</a>
15:17:37 Triggering projects: [post-build-status-update](https://ci.nodejs.org/job/post-build-status-update/)
15:17:37     >> Job status: [node-test-commit] the 'build only if scm changes' feature is disabled.
15:17:37 Starting build job [node-test-commit](https://ci.nodejs.org/job/node-test-commit/) at 10:17:37.

We should endeavour to understand whether queuing up here would negatively impact the trigger code by potentially making it think the PR test had not been initated successfully if the error message is not quickly published to the originating issue. If that turns out to be a problem then we should throttle node-test-commit instead, although that still comes with a risk of node-test-pull-requests filling up the executors when a problem such as today's occurs (just now all of its subjobs)