Skip to content

ci: Windows "Verify global install from pack" takes 8-13m and intermittently blows the 20m job timeout, blocking releases #136

Description

@Rinse12

Symptom

The test-bitsocial-cli (windows-latest) job intermittently hits timeout-minutes: 20 (.github/workflows/CI.yml:39) and is cancelled inside the Verify global install from pack step, before running a single test. When this happens on a master push, CI-build.yml — which is gated on if: github.event.workflow_run.conclusion == 'success' — is skipped, so no release is published.

Most recent occurrence: run 32461656403 (master, 0f6ef94), which silently blocked the 0.0.85 pkc-js release.

It is chronic, not a regression

Step duration for that step across the last 40 CI runs (windows job only):

  • 8-13 minutes, consistently, going back to at least 2026-06-25
  • npm ci — the same 1386-package tree, same runner, same job — takes 2.3 minutes
  • the entire test suite takes 2.5 minutes
  • total job is typically 13-19m against a 20m cap

Timeouts so far: 2026-07-30 (0382b13, step 17.1m), 2026-08-13 (53b42e5, 15.8m), 2026-08-21 (0f6ef94).

The margin is thin enough that any run can land on either side of the line, so releases fail to publish at random.

The existing mitigation does not work

ff6aa9b ci: speed up global install with --prefer-offline was added immediately after the 2026-07-30 timeout, and the in-workflow comment credits it with fixing a 17m step. The measurements do not support that:

  • median step time before ff6aa9b: ~10.0m
  • median step time after ff6aa9b: ~10.7m

The 17.1m was the tail of an already-noisy 8-13m distribution; the flag was credited with a regression to the mean.

Root cause

npm install -g <tarball> has no lockfile, so it re-resolves every dependency range from scratch. That defeats the cache npm ci primes, in two independent ways:

  1. No cached metadata. Resolution needs a packument for each of ~1386 packages. npm ci never fetches packuments — it reads resolved URLs directly from the lockfile — so the cache it primes holds tarballs but no metadata. --prefer-offline therefore has nothing to reuse and goes to the network for the whole tree.
  2. Resolved versions drift from the lockfile. Diffing what the global install resolves against package-lock.json: 243 of 1386 packages resolve to a different version (e.g. axios 1.16.0 -> 1.19.0, agent-base 7.1.4 -> 6.0.2, most of @babel/*). Those tarballs are cache misses too.

Log profile of a representative 12.2m run (windows job 96705351391) supports this split:

  • ~6m spread across the deprecation warnings — resolution
  • a single 6.3m gap with no output at all — reify (extraction into the Windows global prefix, plus postinstall scripts)

For reference, resolution alone with warm metadata takes 8 seconds locally, so the cost is network round-trips and file I/O, not CPU.

Plan

Measure before changing anything — a windows-only workflow_dispatch matrix timing several variants in one run:

  • baseline — current flags (control)
  • npm 12.0.2 — the workflow currently pins corepack prepare npm@11.13.0
  • --ignore-scripts — isolates how much of the 6.3m reify gap is postinstall scripts (kubo binary download, native prebuilds) vs. extraction
  • --offline — hard cache-only; quantifies exactly how much of the cost is network

Then implement whichever the numbers justify, and only afterwards revisit timeout-minutes.

Raising the timeout alone is explicitly not the fix: a step that takes 10 minutes to install a tree npm ci installs in 2.3 is the actual defect.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions