diff --git a/.github/scripts/README.md b/.github/scripts/README.md index b308409..aa1a511 100644 --- a/.github/scripts/README.md +++ b/.github/scripts/README.md @@ -78,12 +78,69 @@ on the numeric base of the Spring Boot version in the release's properties file, floor set by the *phase* rather than that Boot version verbatim, so the bound does not churn on every milestone. +## `version-cmp.js` + +`cmp(a, b)`: dotted-numeric version comparison, tolerant of a leading `v` (release tags like +`v0.4.26`). Shared because this exact comparator used to be implemented three times +independently — `update-maven-wrapper.yml`'s `versions` job, `maven-wrapper-properties.js`'s +own `cmp` (now a re-export of this module), and what would otherwise be a fourth copy for the +Antora UI bundle's release tags in `antora-ui-bundle.js`. + +## `gh-cli.js` + +`gh(args)`, `ghRetry(args, attempts)`, `ghJson(path, method, payload)`: the `execFileSync('gh', +...)` wrapper, retry-with-backoff, and write-payload-then-`--input` helpers every workflow +that fans out `gh api` calls across many repositories needs. `ghJson` writes the payload to +`payload.json` rather than passing it with repeated `-f` flags, because `-f` cannot express +the nested `author`/`committer` objects the git data API needs and would mangle newlines in a +commit message or PR body. + +## `git-pr-helpers.js` + +The generic git-data-API and pull-request primitives shared by `update-maven-wrapper.yml` and +`update-antora-ui-bundle.yml`, both of which open a bot PR the same way: `readFileAt(repo, +path, ref)` (contents API, base64-decoded), `createBranch(repo, headRef, baseBranch)`, +`commitFilesToRef(repo, ref, files, message, author)` (one commit for every file via the git +data API's tree/commit/ref-move sequence, rather than the contents API's one-file-one-commit +shape), `findOpenPrByHeadPrefix(repo, base, headPrefix)` (matched by prefix rather than exact +head name, so a PR opened for an earlier target is found and bumped in place instead of +stacking a second PR), and `openPr(repo, { title, head, base, body })`. Built on `gh-cli.js`. +What differs per workflow — deciding *whether* something changed, and what the branch/PR +naming and body text say — stays local to each workflow's own script. + +## `merge-if-green.js` + +`classifyChecks(statusCheckRollup)` and `mergeIfGreen({ repo, pr, method, dryRun })`: reads a +PR's mergeability and checks, and merges it only when every check has reported and passed, +and GitHub reports a `CLEAN` merge state (not merely `MERGEABLE`, which also covers a locked +branch or a pending required review). Checks are read from `statusCheckRollup` rather than +inferred from `mergeStateStatus`, which conflates failing checks with "needs review" and a +locked branch — the same distinction `dependabot-report.yml`'s own PR-state classification +makes independently, in its own `dependabot-scan` action, for Dependabot-authored PRs (a +close cousin of this module, not yet unified with it). Shared by the "Merge PR if all checks +pass" step of `update-maven-wrapper.yml` and `update-antora-ui-bundle.yml`. + +## `project-branch-matrix.js` + +The `config/projects.json` traversal shared by every workflow that fans out across Spring +Cloud's OSS and commercial repositories: `parseProjectFilter`, `typeKeysFor`, `repoName`, +`knownProjectKeys`, and the underlying `walkProjects` walk, plus two convenience builders for +the two matrix shapes already in use across the estate — `buildBranchMatrix` (one entry per +branch, used by `update-maven-wrapper.yml`, `update-antora-ui-bundle.yml`, +`ci-status-report.yml` and `rollout-actions-ref.yml`) and `buildRepoMatrix` (one entry per +repository with its branches comma-joined, used by `dependabot-report.yml`, +`dependabot-triage.yml` and `lock-unlock-branches.yml`). Before this was extracted, six +workflows each carried their own copy of the same read-filter-walk logic, differing only in +how they turned a project's branches into matrix entries — that's the only part left to each +caller now. + ## `maven-wrapper-properties.js` The rules by which `update-maven-wrapper.yml` edits `maven-wrapper.properties`. It is shared because that workflow edits those files from two different places — through the GitHub contents/git APIs in properties-only mode, and against a real checkout in regenerate mode — -and the two must produce byte-identical results. +and the two must produce byte-identical results. Version comparison is delegated to +`version-cmp.js` (`cmp` here is a re-export) rather than duplicated. It also carries a port of Dependabot's own wrapper-version resolution (`maven/lib/dependabot/maven/file_parser/wrapper_mojo.rb`): `wrapperVersion`, then a version @@ -93,6 +150,16 @@ repository's entire update job — no pull requests at all, not merely no wrappe what the workflow's `check_only` mode predicts, and matching Dependabot's logic exactly is why it can. +## `antora-ui-bundle.js` + +The rules by which `update-antora-ui-bundle.yml` edits `docs/antora-playbook.yml`: +`PLAYBOOK_PATH`, `extractBundle(text)` (reads the `{ repo, tag, url }` of the UI bundle +release a playbook currently points at, matched by shape — a GitHub release asset download +link — rather than by position under `ui:`/`bundle:`/`url:`), and `rewrite(text, { repo, tag +})` (a textual, non-destructive URL swap — everything else in the file is preserved +byte-for-byte, the same contract `maven-wrapper-properties.js`'s `rewrite` gives for +`maven-wrapper.properties`). `cmp` is re-exported from `version-cmp.js` for tag comparison. + ## Tests ```bash @@ -101,10 +168,10 @@ npm install npm test # or: npm run test:coverage ``` -CI runs them in [test-maven-wrapper-properties.yml](../workflows/test-maven-wrapper-properties.yml), -which also extracts every inline `node` heredoc from `update-maven-wrapper.yml` and -syntax-checks it — a syntax error inside a YAML heredoc is otherwise invisible until the -workflow runs against a real repository. +CI runs them in [test-shared-scripts.yml](../workflows/test-shared-scripts.yml), which also +extracts every inline `node` heredoc from each consuming workflow and syntax-checks it — a +syntax error inside a YAML heredoc is otherwise invisible until the workflow runs against a +real repository. Every module here is covered; `npm test` runs the lot. A workflow that `require`s one of them **must check the repository out first** — these resolve under diff --git a/.github/scripts/__tests__/antora-ui-bundle.test.js b/.github/scripts/__tests__/antora-ui-bundle.test.js new file mode 100644 index 0000000..8eb2c8b --- /dev/null +++ b/.github/scripts/__tests__/antora-ui-bundle.test.js @@ -0,0 +1,63 @@ +const { PLAYBOOK_PATH, extractBundle, rewrite, cmp } = require('../antora-ui-bundle'); + +// The exact shape confirmed live on spring-cloud-commons's main, 4.2.x, 4.3.x and 5.0.x +// branches (and their -commercial counterparts). +const PLAYBOOK = [ + 'antora:', + ' extensions:', + " - require: '@springio/antora-extensions'", + " root_component_name: 'cloud-commons'", + 'site:', + ' title: Spring Cloud Commons', + ' url: https://docs.spring.io/spring-cloud-commons/reference/', + 'content:', + ' sources:', + ' - url: ./..', + ' branches: HEAD', + ' start_path: docs', + ' worktrees: true', + 'ui:', + ' bundle:', + ' url: https://github.com/spring-io/antora-ui-spring/releases/download/v0.4.15/ui-bundle.zip', + '', +].join('\n'); + +describe('PLAYBOOK_PATH', () => { + it('is the per-branch playbook path', () => { + expect(PLAYBOOK_PATH).toBe('docs/antora-playbook.yml'); + }); +}); + +describe('extractBundle', () => { + it('reads the repo, tag and full url from a real playbook', () => { + expect(extractBundle(PLAYBOOK)).toEqual({ + repo: 'spring-io/antora-ui-spring', + tag: 'v0.4.15', + url: 'https://github.com/spring-io/antora-ui-spring/releases/download/v0.4.15/ui-bundle.zip', + }); + }); + + it('returns null when there is no release-download url at all', () => { + expect(extractBundle('ui:\n bundle:\n url: https://example.com/ui-bundle.zip\n')).toBeNull(); + }); +}); + +describe('rewrite', () => { + it('replaces only the tag, leaving everything else byte-identical', () => { + const out = rewrite(PLAYBOOK, { repo: 'spring-io/antora-ui-spring', tag: 'v0.4.26' }); + expect(out).toContain( + 'url: https://github.com/spring-io/antora-ui-spring/releases/download/v0.4.26/ui-bundle.zip'); + expect(out.replace('v0.4.26', 'v0.4.15')).toBe(PLAYBOOK); + }); + + it('is a no-op when the file has no bundle url', () => { + const noBundle = 'site:\n title: X\n'; + expect(rewrite(noBundle, { repo: 'spring-io/antora-ui-spring', tag: 'v0.4.26' })).toBe(noBundle); + }); +}); + +describe('cmp', () => { + it('is re-exported from version-cmp for tag comparison', () => { + expect(cmp('v0.4.26', 'v0.4.15')).toBeGreaterThan(0); + }); +}); diff --git a/.github/scripts/__tests__/merge-if-green.test.js b/.github/scripts/__tests__/merge-if-green.test.js new file mode 100644 index 0000000..1432d8e --- /dev/null +++ b/.github/scripts/__tests__/merge-if-green.test.js @@ -0,0 +1,66 @@ +const { classifyChecks } = require('../merge-if-green'); + +// Fixture shapes lifted from what `gh pr view --json statusCheckRollup` actually returns: +// a mix of legacy commit statuses (StatusContext) and modern check runs (CheckRun). +const checkRun = (name, status, conclusion) => ({ __typename: 'CheckRun', name, status, conclusion }); +const statusContext = (context, state) => ({ __typename: 'StatusContext', context, state }); + +describe('classifyChecks', () => { + it('treats an empty rollup as no checks', () => { + const { checks, failing, pending } = classifyChecks([]); + expect(checks).toEqual([]); + expect(failing).toEqual([]); + expect(pending).toEqual([]); + }); + + it('treats undefined as no checks', () => { + expect(classifyChecks(undefined).checks).toEqual([]); + }); + + it('classifies a completed, successful check run as passing', () => { + const { failing, pending } = classifyChecks([checkRun('build', 'COMPLETED', 'SUCCESS')]); + expect(failing).toEqual([]); + expect(pending).toEqual([]); + }); + + it('classifies an in-progress check run as pending', () => { + const { pending } = classifyChecks([checkRun('build', 'IN_PROGRESS', null)]); + expect(pending.map(c => c.name)).toEqual(['build']); + }); + + it('classifies a failed check run as failing', () => { + const { failing } = classifyChecks([checkRun('build', 'COMPLETED', 'FAILURE')]); + expect(failing.map(c => c.name)).toEqual(['build']); + }); + + it('classifies NEUTRAL and SKIPPED conclusions as passing, not pending', () => { + const { failing, pending } = classifyChecks([ + checkRun('optional', 'COMPLETED', 'NEUTRAL'), + checkRun('skipped-job', 'COMPLETED', 'SKIPPED'), + ]); + expect(failing).toEqual([]); + expect(pending).toEqual([]); + }); + + it('classifies a legacy status context by its state', () => { + const { failing, pending } = classifyChecks([statusContext('ci/legacy', 'SUCCESS')]); + expect(failing).toEqual([]); + expect(pending).toEqual([]); + }); + + it('classifies a pending legacy status context as pending', () => { + const { pending } = classifyChecks([statusContext('ci/legacy', 'PENDING')]); + expect(pending.map(c => c.name)).toEqual(['ci/legacy']); + }); + + it('separates failing and pending checks in a mixed rollup', () => { + const { checks, failing, pending } = classifyChecks([ + checkRun('build', 'COMPLETED', 'SUCCESS'), + checkRun('test', 'COMPLETED', 'FAILURE'), + checkRun('lint', 'IN_PROGRESS', null), + ]); + expect(checks).toHaveLength(3); + expect(failing.map(c => c.name)).toEqual(['test']); + expect(pending.map(c => c.name)).toEqual(['lint']); + }); +}); diff --git a/.github/scripts/__tests__/project-branch-matrix.test.js b/.github/scripts/__tests__/project-branch-matrix.test.js new file mode 100644 index 0000000..35dfb57 --- /dev/null +++ b/.github/scripts/__tests__/project-branch-matrix.test.js @@ -0,0 +1,173 @@ +const { + parseProjectFilter, typeKeysFor, repoName, knownProjectKeys, + walkProjects, buildBranchMatrix, buildRepoMatrix, +} = require('../project-branch-matrix'); + +// A trimmed fixture shaped like the real config/projects.json: two OSS-and-commercial +// projects, one commercial-only project (mirroring spring-cloud-sleuth), and a `defaults` +// key that must never be treated as a project. +const PROJECTS = { + 'spring-cloud-alpha': { + oss: { branches: { scheduled: ['main', '5.0.x'], default: ['main'] } }, + commercial: { + branches: { + scheduled: ['5.1.x-internal', '4.3.x', '4.2.x'], + default: ['4.3.x'], + }, + }, + }, + 'spring-cloud-beta': { + oss: { branches: { scheduled: ['main'], default: ['main'] } }, + commercial: { branches: { scheduled: ['3.1.x'], default: ['3.1.x'] } }, + }, + 'spring-cloud-commercial-only': { + commercial: { branches: { scheduled: ['3.1.x'], default: ['3.1.x'] } }, + }, + defaults: { + oss: { branches: { scheduled: ['main'], default: ['main'] } }, + }, +}; + +describe('parseProjectFilter', () => { + it('returns an empty set for blank input', () => { + expect(parseProjectFilter('').size).toBe(0); + expect(parseProjectFilter(undefined).size).toBe(0); + }); + + it('splits, trims and drops empties from a comma-separated list', () => { + expect(parseProjectFilter(' spring-cloud-alpha ,, spring-cloud-beta,')) + .toEqual(new Set(['spring-cloud-alpha', 'spring-cloud-beta'])); + }); +}); + +describe('typeKeysFor', () => { + it('expands both/empty to oss and commercial', () => { + expect(typeKeysFor('both')).toEqual(['oss', 'commercial']); + expect(typeKeysFor('')).toEqual(['oss', 'commercial']); + expect(typeKeysFor(undefined)).toEqual(['oss', 'commercial']); + }); + + it('passes a single type straight through', () => { + expect(typeKeysFor('oss')).toEqual(['oss']); + expect(typeKeysFor('commercial')).toEqual(['commercial']); + }); +}); + +describe('repoName', () => { + it('builds the plain repo name for oss', () => { + expect(repoName('spring-cloud-alpha', 'oss')).toBe('spring-cloud/spring-cloud-alpha'); + }); + + it('appends -commercial for commercial', () => { + expect(repoName('spring-cloud-alpha', 'commercial')) + .toBe('spring-cloud/spring-cloud-alpha-commercial'); + }); +}); + +describe('knownProjectKeys', () => { + it('lists every project except defaults', () => { + expect(knownProjectKeys(PROJECTS).sort()).toEqual([ + 'spring-cloud-alpha', 'spring-cloud-beta', 'spring-cloud-commercial-only', + ]); + }); +}); + +describe('walkProjects', () => { + it('skips defaults and visits every project/type pair present', () => { + const seen = []; + walkProjects(PROJECTS, {}, ({ projectKey, typeKey }) => seen.push(`${projectKey}:${typeKey}`)); + expect(seen).toEqual(expect.arrayContaining([ + 'spring-cloud-alpha:oss', 'spring-cloud-alpha:commercial', + 'spring-cloud-beta:oss', 'spring-cloud-beta:commercial', + 'spring-cloud-commercial-only:commercial', + ])); + expect(seen).not.toContain('defaults:oss'); + expect(seen).not.toContain('spring-cloud-commercial-only:oss'); + }); + + it('applies a project filter', () => { + const seen = []; + walkProjects(PROJECTS, { filter: 'spring-cloud-alpha' }, ({ projectKey }) => seen.push(projectKey)); + expect(new Set(seen)).toEqual(new Set(['spring-cloud-alpha'])); + }); + + it('applies a repoType filter', () => { + const seen = []; + walkProjects(PROJECTS, { repoType: 'oss' }, ({ typeKey }) => seen.push(typeKey)); + expect(new Set(seen)).toEqual(new Set(['oss'])); + }); +}); + +describe('buildBranchMatrix', () => { + it('builds one entry per scheduled branch and excludes -internal by default', () => { + const { entries, excluded } = buildBranchMatrix(PROJECTS, { repoType: 'commercial' }); + expect(entries).toEqual(expect.arrayContaining([ + { project: 'spring-cloud-alpha', repo: 'spring-cloud/spring-cloud-alpha-commercial', type: 'commercial', branch: '4.2.x' }, + { project: 'spring-cloud-alpha', repo: 'spring-cloud/spring-cloud-alpha-commercial', type: 'commercial', branch: '4.3.x' }, + ])); + expect(entries.some(e => e.branch === '5.1.x-internal')).toBe(false); + expect(excluded).toEqual(['spring-cloud/spring-cloud-alpha-commercial@5.1.x-internal']); + }); + + it('keeps -internal branches in entries when excludeInternal is false', () => { + const { entries, excluded } = buildBranchMatrix(PROJECTS, { + repoType: 'commercial', excludeInternal: false, + }); + expect(entries.some(e => e.branch === '5.1.x-internal')).toBe(true); + expect(excluded).toEqual([]); + }); + + it('sorts by repo then branch', () => { + // Sorted by the repo/branch *fields* via localeCompare, not by the concatenated + // "repo@branch" string - those disagree here, since '-' sorts before '@' and + // "spring-cloud-beta" is a prefix of "spring-cloud-beta-commercial". + const { entries } = buildBranchMatrix(PROJECTS, { filter: 'spring-cloud-beta', repoType: 'both' }); + expect(entries.map(e => `${e.repo}@${e.branch}`)).toEqual([ + 'spring-cloud/spring-cloud-beta@main', + 'spring-cloud/spring-cloud-beta-commercial@3.1.x', + ]); + for (let i = 1; i < entries.length; i++) { + const cmp = entries[i - 1].repo.localeCompare(entries[i].repo) || + entries[i - 1].branch.localeCompare(entries[i].branch); + expect(cmp).toBeLessThanOrEqual(0); + } + }); +}); + +describe('buildRepoMatrix', () => { + it('builds one entry per repo with branches joined by comma', () => { + const { entries } = buildRepoMatrix(PROJECTS, { filter: 'spring-cloud-beta' }); + expect(entries).toEqual(expect.arrayContaining([ + { project: 'spring-cloud-beta', repo: 'spring-cloud/spring-cloud-beta', type: 'oss', branches: 'main' }, + { project: 'spring-cloud-beta', repo: 'spring-cloud/spring-cloud-beta-commercial', type: 'commercial', branches: '3.1.x' }, + ])); + }); + + it('omits a repo whose type section is absent entirely', () => { + const { entries } = buildRepoMatrix(PROJECTS, { + filter: 'spring-cloud-commercial-only', repoType: 'oss', + }); + expect(entries).toEqual([]); + }); + + it('still includes a repo whose section has an empty branch list, unlike buildBranchMatrix', () => { + // dependabot-report.yml and dependabot-triage.yml always push one entry per surviving + // (project, type) pair, even with nothing to scan - only lock-unlock-branches.yml wants + // that case dropped, and it filters the result itself rather than this function deciding + // for every caller. + const projects = { empty: { oss: { branches: { scheduled: [] } } } }; + const { entries } = buildRepoMatrix(projects, { repoType: 'oss' }); + expect(entries).toEqual([{ project: 'empty', repo: 'spring-cloud/empty', type: 'oss', branches: '' }]); + }); + + it('supports a custom branchesOf selector (scheduled union default)', () => { + const { entries } = buildRepoMatrix(PROJECTS, { + filter: 'spring-cloud-alpha', + repoType: 'commercial', + branchesOf: section => + [...new Set([...(section.branches?.scheduled || []), ...(section.branches?.default || [])])], + }); + expect(entries[0].branches.split(',').sort()).toEqual( + ['4.2.x', '4.3.x', '5.1.x-internal'].sort()); + }); +}); diff --git a/.github/scripts/__tests__/version-cmp.test.js b/.github/scripts/__tests__/version-cmp.test.js new file mode 100644 index 0000000..8d6ec18 --- /dev/null +++ b/.github/scripts/__tests__/version-cmp.test.js @@ -0,0 +1,21 @@ +const { cmp } = require('../version-cmp'); + +describe('cmp', () => { + it('orders by major, then minor, then patch', () => { + expect(cmp('3.9.16', '3.9.9')).toBeGreaterThan(0); + expect(cmp('3.9.9', '3.9.16')).toBeLessThan(0); + expect(cmp('3.9.16', '3.9.16')).toBe(0); + expect(cmp('4.0.0', '3.9.16')).toBeGreaterThan(0); + }); + + it('treats a missing trailing segment as 0', () => { + expect(cmp('3.9', '3.9.0')).toBe(0); + expect(cmp('3.9.1', '3.9')).toBeGreaterThan(0); + }); + + it('strips a leading v from either side', () => { + expect(cmp('v0.4.26', 'v0.4.15')).toBeGreaterThan(0); + expect(cmp('v0.4.15', '0.4.15')).toBe(0); + expect(cmp('0.4.26', 'v0.4.15')).toBeGreaterThan(0); + }); +}); diff --git a/.github/scripts/antora-ui-bundle.js b/.github/scripts/antora-ui-bundle.js new file mode 100644 index 0000000..dec5683 --- /dev/null +++ b/.github/scripts/antora-ui-bundle.js @@ -0,0 +1,36 @@ +'use strict'; + +const { cmp } = require('./version-cmp'); + +// docs/antora-playbook.yml is per-branch (content.sources[].url: ./.. with branches: HEAD - +// it documents its own branch), so every maintained branch carries its own copy rather than +// there being one canonical file on main. Branches that predate Antora adoption have no +// such file at all. +const PLAYBOOK_PATH = 'docs/antora-playbook.yml'; + +// Matches the UI bundle download URL wherever it appears in the file - matched by shape (a +// GitHub release asset download link) rather than by indentation under ui:/bundle:/url:, +// the same way currentMaven() in maven-wrapper-properties.js matches distributionUrl by +// shape rather than by position. +const BUNDLE_URL = + /https:\/\/github\.com\/([^/\s]+\/[^/\s]+)\/releases\/download\/([^/\s]+)\/([^\s'"]+)/; + +// { repo, tag, url } for the UI bundle release this playbook currently points at, or null +// when no release-download URL is present at all. +function extractBundle(text) { + const m = text.match(BUNDLE_URL); + if (!m) return null; + return { repo: m[1], tag: m[2], url: m[0] }; +} + +// Textual, non-destructive: only the URL is replaced, everything else in the file - +// comments, key order, unrelated settings - is preserved byte-for-byte. Mirrors the +// contract maven-wrapper-properties.js's rewrite() gives for maven-wrapper.properties. +function rewrite(text, { repo, tag }) { + const current = extractBundle(text); + if (!current) return text; + const newUrl = current.url.replace(current.repo, repo).replace(current.tag, tag); + return text.replace(current.url, newUrl); +} + +module.exports = { PLAYBOOK_PATH, extractBundle, rewrite, cmp }; diff --git a/.github/scripts/gh-cli.js b/.github/scripts/gh-cli.js new file mode 100644 index 0000000..08b107f --- /dev/null +++ b/.github/scripts/gh-cli.js @@ -0,0 +1,45 @@ +'use strict'; + +const fs = require('fs'); +const { execFileSync } = require('child_process'); + +// Runs the gh CLI, capturing stdout/stderr instead of letting a failure throw all the way +// out - a script running from a YAML heredoc has no caller that could do anything useful +// with an uncaught exception besides a worse stack trace. +function gh(args) { + try { + return { + ok: true, + out: execFileSync('gh', args, + { encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'], maxBuffer: 1 << 26 }), + }; + } catch (err) { + return { + ok: false, + out: err.stdout || '', + err: (err.stderr || err.message || '').split('\n')[0].trim(), + }; + } +} + +// A transient gh/API failure (rate limit, network blip) shouldn't fail a whole matrix leg, +// so a failed attempt is retried with a short linear backoff before giving up. +function ghRetry(args, attempts = 3) { + let last; + for (let i = 1; i <= attempts; i++) { + last = gh(args); + if (last.ok) return last; + if (i < attempts) execFileSync('sleep', [String(i * 3)]); + } + return last; +} + +// Write requests go through --input with a JSON file rather than repeated -f flags: -f +// cannot express the nested author/committer objects the git data API needs, and it would +// mangle the newlines in a commit message or PR body. +function ghJson(path, method, payload) { + fs.writeFileSync('payload.json', JSON.stringify(payload)); + return ghRetry(['api', path, '--method', method, '--input', 'payload.json']); +} + +module.exports = { gh, ghRetry, ghJson }; diff --git a/.github/scripts/git-pr-helpers.js b/.github/scripts/git-pr-helpers.js new file mode 100644 index 0000000..0a5b253 --- /dev/null +++ b/.github/scripts/git-pr-helpers.js @@ -0,0 +1,80 @@ +'use strict'; + +const { gh, ghRetry, ghJson } = require('./gh-cli'); + +// Contents API read, base64-decoded. Returns null on any failure (404, unknown ref, ...) +// rather than throwing - callers treat "not found" as meaningful, not exceptional. +function readFileAt(repo, path, ref) { + const r = gh(['api', `repos/${repo}/contents/${path}?ref=${encodeURIComponent(ref)}`]); + if (!r.ok) return null; + const meta = JSON.parse(r.out); + return { sha: meta.sha, text: Buffer.from(meta.content, 'base64').toString('utf8') }; +} + +// Creates headRef from the current tip of baseBranch. `alreadyExists` distinguishes +// "someone else's leftover branch with no PR" (the caller decides what to do about that) +// from a hard failure to create it. +function createBranch(repo, headRef, baseBranch) { + const baseRef = ghRetry(['api', `repos/${repo}/git/ref/heads/${baseBranch}`, '--jq', '.object.sha']); + if (!baseRef.ok) return { ok: false, err: `could not read ${baseBranch}: ${baseRef.err}` }; + + const made = ghJson(`repos/${repo}/git/refs`, 'POST', + { ref: `refs/heads/${headRef}`, sha: baseRef.out.trim() }); + if (made.ok) return { ok: true, alreadyExists: false }; + if (/already exists/i.test(made.err)) return { ok: true, alreadyExists: true }; + return { ok: false, err: `could not create ${headRef}: ${made.err}` }; +} + +// One commit for every file in `files` (each `{ path, content }`), via the git data API's +// tree/commit/ref-move sequence rather than the contents API's one-file-one-commit shape - +// a branch touching N files should read as one commit, not N sharing the same message. +function commitFilesToRef(repo, ref, files, message, author) { + const refSha = ghRetry(['api', `repos/${repo}/git/ref/heads/${ref}`, '--jq', '.object.sha']); + if (!refSha.ok) return { ok: false, err: `could not read ${ref}: ${refSha.err}` }; + const parent = refSha.out.trim(); + + const baseTree = ghRetry(['api', `repos/${repo}/git/commits/${parent}`, '--jq', '.tree.sha']); + if (!baseTree.ok) return { ok: false, err: `could not read the tree of ${ref}: ${baseTree.err}` }; + + // mode 100644: every file this is used for (maven-wrapper.properties, antora-playbook.yml) + // is never executable. Everything not listed is inherited from base_tree. + const made = ghJson(`repos/${repo}/git/trees`, 'POST', { + base_tree: baseTree.out.trim(), + tree: files.map(f => ({ path: f.path, mode: '100644', type: 'blob', content: f.content })), + }); + if (!made.ok) return { ok: false, err: `could not build a tree: ${made.err}` }; + + const commit = ghJson(`repos/${repo}/git/commits`, 'POST', { + message, + tree: JSON.parse(made.out).sha, + parents: [parent], + author, + committer: author, + }); + if (!commit.ok) return { ok: false, err: `could not create a commit: ${commit.err}` }; + + const moved = ghJson(`repos/${repo}/git/refs/heads/${ref}`, 'PATCH', + { sha: JSON.parse(commit.out).sha }); + if (!moved.ok) return { ok: false, err: `could not move ${ref}: ${moved.err}` }; + return { ok: true }; +} + +// Matched by prefix rather than exact head name, so a PR opened for an earlier target (an +// older Maven release, an older UI bundle tag) is found and bumped in place instead of +// stacking a second PR on top of it. +function findOpenPrByHeadPrefix(repo, base, headPrefix) { + const enc = encodeURIComponent(base); + const openPrs = ghRetry(['api', `repos/${repo}/pulls?state=open&base=${enc}&per_page=100`]); + if (!openPrs.ok) return { ok: false, err: `could not list pull requests: ${openPrs.err}` }; + const mine = JSON.parse(openPrs.out || '[]') + .filter(pr => (pr.head?.ref || '').startsWith(headPrefix)); + return { ok: true, pr: mine[0] || null }; +} + +function openPr(repo, { title, head, base, body }) { + const pr = ghJson(`repos/${repo}/pulls`, 'POST', { title, head, base, body }); + if (!pr.ok) return { ok: false, err: `could not open PR: ${pr.err}` }; + return { ok: true, pr: JSON.parse(pr.out) }; +} + +module.exports = { readFileAt, createBranch, commitFilesToRef, findOpenPrByHeadPrefix, openPr }; diff --git a/.github/scripts/maven-wrapper-properties.js b/.github/scripts/maven-wrapper-properties.js index bc4a668..69fa500 100644 --- a/.github/scripts/maven-wrapper-properties.js +++ b/.github/scripts/maven-wrapper-properties.js @@ -11,6 +11,8 @@ // the repository - no pull requests at all, not merely no wrapper PR. Matching its logic is // what lets `check_only` predict that failure before Dependabot hits it. +const { cmp } = require('./version-cmp'); + const PROPS_SUFFIX = '.mvn/wrapper/maven-wrapper.properties'; // Scripts Dependabot will read a version banner out of, in its own preference order: @@ -42,14 +44,6 @@ const SCRIPT_VERSION = const propsPath = dir => (dir === '.' ? PROPS_SUFFIX : `${dir}/${PROPS_SUFFIX}`); const scriptPaths = dir => SCRIPTS.map(s => (dir === '.' ? s : `${dir}/${s}`)); -const cmp = (a, b) => { - const pa = String(a).split('.').map(Number), pb = String(b).split('.').map(Number); - for (let i = 0; i < Math.max(pa.length, pb.length); i++) { - if ((pa[i] || 0) !== (pb[i] || 0)) return (pa[i] || 0) - (pb[i] || 0); - } - return 0; -}; - // Java .properties semantics, matched to Dependabot's get_property_value: line // continuations are joined first, `#` and `!` start comments, and either `=` or `:` // separates key from value. A hand-rolled /^key=/m would disagree with Dependabot on diff --git a/.github/scripts/merge-if-green.js b/.github/scripts/merge-if-green.js new file mode 100644 index 0000000..ab86f46 --- /dev/null +++ b/.github/scripts/merge-if-green.js @@ -0,0 +1,62 @@ +'use strict'; + +const { gh } = require('./gh-cli'); + +const FAILING = new Set(['FAILURE', 'TIMED_OUT', 'ERROR', 'STARTUP_FAILURE', 'CANCELLED']); +const PASSING = new Set(['SUCCESS', 'NEUTRAL', 'SKIPPED']); + +// Normalizes the two shapes `statusCheckRollup` can contain - legacy commit statuses and +// modern check runs - into one `{ name, state }` list, then splits it into failing/pending +// so a caller can decide, or explain, whether a PR is safe to merge. +function classifyChecks(statusCheckRollup) { + const checks = (statusCheckRollup || []).map(c => { + if (c.__typename === 'StatusContext' || c.state) { + return { name: c.context || 'status', state: (c.state || '').toUpperCase() }; + } + const status = (c.status || '').toUpperCase(); + if (status && status !== 'COMPLETED') return { name: c.name, state: 'PENDING' }; + return { name: c.name, state: (c.conclusion || 'PENDING').toUpperCase() }; + }); + const failing = checks.filter(c => FAILING.has(c.state)); + const pending = checks.filter(c => !FAILING.has(c.state) && !PASSING.has(c.state)); + return { checks, failing, pending }; +} + +// Reads a PR's mergeability and checks, and merges it if - and only if - every check has +// reported and passed, GitHub reports a CLEAN merge state (not merely "mergeable", which +// also covers a locked branch or a pending required review), and dryRun is false. Returns +// `{ status, detail }` rather than exiting the process - the caller writes its own result +// artifact and decides what "not-merged" should be called in its own summary. +function mergeIfGreen({ repo, pr, method = 'squash', dryRun = true }) { + const view = gh(['pr', 'view', String(pr), '--repo', repo, '--json', + 'mergeable,mergeStateStatus,statusCheckRollup,url']); + if (!view.ok) return { status: 'error', detail: `could not read PR: ${view.err}` }; + + const data = JSON.parse(view.out); + const { checks, failing, pending } = classifyChecks(data.statusCheckRollup); + + if (!checks.length) return { status: 'not-merged', detail: 'no checks have reported yet' }; + if (failing.length) { + return { status: 'not-merged', detail: `failing checks: ${failing.map(c => c.name).join(', ')}` }; + } + if (pending.length) { + return { status: 'not-merged', detail: `checks still running: ${pending.map(c => c.name).join(', ')}` }; + } + if (data.mergeable === 'CONFLICTING') { + return { status: 'not-merged', detail: 'conflicts with the base branch' }; + } + if (data.mergeable !== 'MERGEABLE') { + return { status: 'not-merged', detail: `mergeable is ${data.mergeable}` }; + } + if (data.mergeStateStatus !== 'CLEAN') { + return { status: 'not-merged', detail: `all checks pass but GitHub reports ${data.mergeStateStatus}` }; + } + + if (dryRun) return { status: 'would-merge', detail: `all ${checks.length} check(s) pass` }; + + const merged = gh(['pr', 'merge', String(pr), '--repo', repo, `--${method}`, '--delete-branch']); + if (!merged.ok) return { status: 'error', detail: `merge failed: ${merged.err}` }; + return { status: 'merged', detail: `${method}, all ${checks.length} check(s) passed` }; +} + +module.exports = { classifyChecks, mergeIfGreen }; diff --git a/.github/scripts/project-branch-matrix.js b/.github/scripts/project-branch-matrix.js new file mode 100644 index 0000000..4376f30 --- /dev/null +++ b/.github/scripts/project-branch-matrix.js @@ -0,0 +1,95 @@ +'use strict'; + +// Every consumer of config/projects.json filters by project name and oss/commercial, then +// walks branches.scheduled (or, for a couple of callers, scheduled union default). This +// module factors out exactly that shared traversal - the shape of the resulting matrix +// entries stays with each caller because it genuinely differs: one entry per branch for a +// workflow that hits a per-branch API, one entry per repository (branches joined into a +// string) for a workflow whose API call is repo-wide. + +function parseProjectFilter(raw) { + const trimmed = (raw || '').trim(); + return trimmed + ? new Set(trimmed.split(',').map(p => p.trim()).filter(Boolean)) + : new Set(); +} + +function typeKeysFor(repoType) { + const type = (repoType || 'both').trim(); + return type === 'both' ? ['oss', 'commercial'] : [type]; +} + +function repoName(projectKey, typeKey) { + return typeKey === 'commercial' + ? `spring-cloud/${projectKey}-commercial` + : `spring-cloud/${projectKey}`; +} + +function knownProjectKeys(projects) { + return Object.keys(projects).filter(k => k !== 'defaults'); +} + +// Visits every (project, type) pair that survives the projects/repo_type filters, in +// projects.json's own order. `visit` receives { projectKey, typeKey, repo, section }, where +// `section` is `projects[projectKey][typeKey]` - the caller decides how to turn that into +// matrix entries. +function walkProjects(projects, { filter, repoType } = {}, visit) { + const filterSet = filter instanceof Set ? filter : parseProjectFilter(filter); + for (const [projectKey, config] of Object.entries(projects)) { + if (projectKey === 'defaults') continue; + if (filterSet.size > 0 && !filterSet.has(projectKey)) continue; + for (const typeKey of typeKeysFor(repoType)) { + const section = config[typeKey]; + if (!section) continue; + visit({ projectKey, typeKey, repo: repoName(projectKey, typeKey), section }); + } + } +} + +// One entry per (repo, branch) drawn from branches.scheduled - the shape +// update-maven-wrapper.yml and ci-status-report.yml already build. `-internal` branches are +// diverted into `excluded` by default, since they are the in-flight development line for the +// next train and most callers leave them alone; pass `excludeInternal: false` to get every +// branch back in `entries` instead (rollout-actions-ref.yml treats internal branches +// differently rather than skipping them, so it needs them present). +function buildBranchMatrix(projects, { filter, repoType, excludeInternal = true } = {}) { + const entries = []; + const excluded = []; + walkProjects(projects, { filter, repoType }, ({ projectKey, typeKey, repo, section }) => { + for (const branch of section.branches?.scheduled || []) { + if (excludeInternal && branch.endsWith('-internal')) { + excluded.push(`${repo}@${branch}`); + continue; + } + entries.push({ project: projectKey, repo, type: typeKey, branch }); + } + }); + entries.sort((a, b) => a.repo.localeCompare(b.repo) || a.branch.localeCompare(b.branch)); + excluded.sort(); + return { entries, excluded }; +} + +// One entry per repository, with its branches joined into a comma-separated string rather +// than fanned out per branch - for callers whose API call is repo-wide (Dependabot PRs are +// listed repo-wide and then attributed to a branch, so a per-branch fan-out would repeat the +// same fetch). A matrix value can't carry an array without `toJson` pretty-printing it and +// breaking the consuming YAML, hence the join. `branchesOf(section)` decides which branches +// count; defaults to branches.scheduled. Every (project, type) pair that survives the +// filters gets an entry, even one with no branches at all - a caller that needs to skip +// those (lock-unlock-branches.yml does, since a repo with nothing to lock is pointless to +// visit) filters the result itself rather than this function deciding for every caller. +function buildRepoMatrix(projects, { filter, repoType, branchesOf } = {}) { + const pick = branchesOf || (section => section.branches?.scheduled || []); + const entries = []; + walkProjects(projects, { filter, repoType }, ({ projectKey, typeKey, repo, section }) => { + const branches = pick(section); + entries.push({ project: projectKey, repo, type: typeKey, branches: branches.join(',') }); + }); + entries.sort((a, b) => a.repo.localeCompare(b.repo)); + return { entries }; +} + +module.exports = { + parseProjectFilter, typeKeysFor, repoName, knownProjectKeys, + walkProjects, buildBranchMatrix, buildRepoMatrix, +}; diff --git a/.github/scripts/version-cmp.js b/.github/scripts/version-cmp.js new file mode 100644 index 0000000..aa4e2c3 --- /dev/null +++ b/.github/scripts/version-cmp.js @@ -0,0 +1,17 @@ +'use strict'; + +// Dotted-numeric version comparison, tolerant of a leading `v` (release tags like +// `v0.4.26`). Shared because this exact comparator was independently implemented three +// times: update-maven-wrapper.yml's `versions` job, maven-wrapper-properties.js's own +// `cmp` (now a re-export of this), and what would otherwise be a fourth copy for the +// Antora UI bundle's release tags. +function cmp(a, b) { + const parts = s => String(s).replace(/^v/, '').split('.').map(Number); + const pa = parts(a), pb = parts(b); + for (let i = 0; i < Math.max(pa.length, pb.length); i++) { + if ((pa[i] || 0) !== (pb[i] || 0)) return (pa[i] || 0) - (pb[i] || 0); + } + return 0; +} + +module.exports = { cmp }; diff --git a/.github/workflows/README-update-antora-ui-bundle.md b/.github/workflows/README-update-antora-ui-bundle.md new file mode 100644 index 0000000..e9981d2 --- /dev/null +++ b/.github/workflows/README-update-antora-ui-bundle.md @@ -0,0 +1,191 @@ +# Update Antora UI Bundle Workflow + +Opens a pull request against every maintained repository/branch whose `docs/antora-playbook.yml` +points at an old release of the Antora UI bundle +([`spring-io/antora-ui-spring`](https://github.com/spring-io/antora-ui-spring)), so **CI +decides whether the bump is safe** rather than nobody noticing the rendered docs look stale. +Defaults to a dry run. + +## Why this exists + +Every Spring Cloud project's docs pin a UI bundle release in `docs/antora-playbook.yml`: + +```yaml +ui: + bundle: + url: https://github.com/spring-io/antora-ui-spring/releases/download/v0.4.15/ui-bundle.zip +``` + +Nobody was watching `spring-io/antora-ui-spring`'s releases, so this drifts silently. At the +time this workflow was written, every maintained branch of `spring-cloud-commons` (and its +`-commercial` counterpart) was pinned to `v0.4.15` while the latest release was already +`v0.4.26` — eleven releases behind. This is the same shape of problem +[`update-maven-wrapper.yml`](README-update-maven-wrapper.md) solves for the Maven wrapper: an +upstream artifact versions independently of every project, and nothing else notices when it +moves. + +## Description + +1. **`versions`** resolves the latest `spring-io/antora-ui-spring` release tag. +2. **`setup`** expands [`config/projects.json`](../../config/projects.json) into one matrix + entry per repo/branch, via the shared + [`project-branch-matrix.js`](../scripts/project-branch-matrix.js). +3. **`update`** compares the branch's UI bundle tag against the target and, if behind, creates + a branch, commits the one-line URL change, and opens a PR. +4. **`summary`** reports every combination in one table, and posts a Google Chat notification + when anything changed. + +## `docs/antora-playbook.yml` is per-branch + +Unlike `maven-wrapper.properties`, this file documents its **own** branch — +`content.sources[].branches` is `HEAD` — so every maintained branch carries its own copy +rather than there being one canonical file on the default branch. Branches that predate +Antora adoption have no such file at all; those are reported as `no-playbook` rather than an +error. + +### What is out of scope + +- **`-internal` branches are skipped**, for the same reason + [`update-maven-wrapper.yml`](README-update-maven-wrapper.md#what-is-out-of-scope) skips + them: they are the in-flight development line for the next train. +- **A repository that ships no `docs/antora-playbook.yml` on any of its branches** never + appears in the summary's table beyond a `no-playbook` row per branch — there is nothing + broken to flag, it simply has not adopted Antora on that branch. + +## What it changes + +Only the UI bundle's release-download URL, via a textual, non-destructive replacement (the +same contract [`maven-wrapper-properties.js`'s `rewrite`](../scripts/README.md) gives for +`maven-wrapper.properties`) — everything else in the file, including comments, key order and +unrelated settings, is preserved byte-for-byte. Applying it to an already-current file is a +no-op. + +A branch already **ahead** of the target (a manual override, or a target resolved before the +branch was last touched) keeps its own tag and is reported as `ahead` — never walked +backwards. + +## Shared with `update-maven-wrapper.yml` + +Rather than reimplementing the same mechanics, this workflow calls the same shared scripts +[`update-maven-wrapper.yml` uses](README-update-maven-wrapper.md): +[`project-branch-matrix.js`](../scripts/project-branch-matrix.js) for the matrix, +[`git-pr-helpers.js`](../scripts/git-pr-helpers.js) for reading a file at a ref, creating a +branch, committing via the git data API, and finding/opening a PR, and +[`merge-if-green.js`](../scripts/merge-if-green.js) for the auto-merge check. Only the +domain-specific rule — how to read and rewrite the UI bundle URL — lives in its own module, +[`antora-ui-bundle.js`](../scripts/antora-ui-bundle.js). + +## DCO + +Spring Cloud repositories run a **required DCO check**, so each commit carries a +`Signed-off-by` trailer and sets the commit author and committer explicitly to match it — +`spring-builds `, same identity +`update-maven-wrapper.yml` uses. + +## Triggers + +- **`workflow_dispatch`** — on demand, with the inputs below. +- **`schedule`** — Mondays at ~10:00am US Eastern, as two month-selected cron entries (EDT + `UTC-4` / EST `UTC-5`), a few hours after `update-maven-wrapper.yml`'s ~7:00am schedule so + the two weekly runs don't land on the shared runner pool at the same moment. + +## Inputs + +| Input | Description | Required | Default | +|-------|-------------|----------|---------| +| `projects` | Comma-separated project names. Empty processes all of them. | No | `''` | +| `repo_type` | `both`, `oss`, or `commercial` | No | `both` | +| `bundle_version` | Target `spring-io/antora-ui-spring` release tag (e.g. `v0.4.26`). Empty uses the latest release. | No | `''` | +| `auto_merge` | Merge existing UI-bundle PRs whose checks have all passed. **Manual runs only** — see [Auto-merge](#auto-merge) | No | `true` | +| `merge_method` | `squash`, `merge`, or `rebase` | No | `squash` | +| `dry_run` | Report what would happen without creating, updating or merging anything | No | `true` | +| `notify` | Post a Google Chat notification when anything changed. Always on for the scheduled run | No | `true` | +| `token` | Needs `contents:write` and `pull-requests:write` on all targets. Falls back to `GH_ACTIONS_REPO_TOKEN`. | No | `''` | + +### What the weekly scheduled run actually does + +A scheduled event carries **no inputs at all**, so every input falls back to what the +expressions decide rather than to the `default:` shown above, exactly as in +[`update-maven-wrapper.yml`](README-update-maven-wrapper.md#what-the-weekly-scheduled-run-actually-does): + +| | Scheduled run | Manual dispatch | +|---|---|---| +| Dry run? | **No — it opens PRs** | Yes by default; uncheck `dry_run` to act | +| Merges green PRs? | **No** | Yes, if `auto_merge` is left checked | +| Notifies Chat? | **Always, if there's anything to report** | Only if `notify` is left checked | +| Scope | Every project, `oss` and `commercial` | As chosen | +| Target version | Latest `spring-io/antora-ui-spring` release | As chosen | +| `-internal` branches | Skipped | Skipped | + +## Auto-merge + +**The weekly scheduled run never merges.** The schedule opens and refreshes PRs, posts the +Chat notification, and leaves the decision to a human. `auto_merge` applies to a manual +dispatch only, making merging every green UI-bundle PR one deliberate click. + +Only a PR that was *already* open when the run started is eligible — one opened or moved up +in the same run has not had CI run against it yet. Checks are read from `statusCheckRollup` +via the shared [`merge-if-green.js`](../scripts/merge-if-green.js), the same module +`update-maven-wrapper.yml` uses — a PR is merged only when every check passes *and* +`mergeable` is `MERGEABLE` *and* `mergeStateStatus` is `CLEAN`. + +| Merge status | Meaning | +|---|---| +| `merged` | Merged and the head branch deleted | +| `would-merge` | Dry run — it would have been merged | +| `not-merged` | Reported with the reason: failing checks (named), checks still running, conflicts, no checks reported yet, or green-but-`BLOCKED` | +| `error` | The merge call itself failed | + +Set `auto_merge` to false to only open and update PRs and leave merging to a human. + +## Google Chat notification + +Posted to `secrets.SPRING_CLOUD_CORE_CI_GCHAT_WEBHOOK_URL` — the same webhook +[`rollout-actions-ref.yml`](README-rollout-actions-ref.md) and +[`dependabot-report.yml`](README-dependabot-report.md) already use for their own +notifications — with the run's summary counts and up to 15 PR links. Skipped silently, with a +line in the job log saying why, when the secret is unset or there is nothing to report: a +weekly "all clear" would only teach people to ignore the channel. + +## Statuses + +| Status | Meaning | +|---|---| +| `pr-opened` | A PR was created | +| `pr-updated` | An open PR was moved up to a newer target — see [Rerunning](#rerunning) | +| `would-open` / `would-update-pr` | Dry run — what would happen | +| `pr-open` | A PR is open and already at the target; nothing done | +| `branch-exists` | The branch exists with no open PR — a previous PR was closed unmerged, so it is **left alone** rather than reopened | +| `up-to-date` | Already on the target | +| `ahead` | Newer than the target; never walked backwards | +| `no-playbook` | No `docs/antora-playbook.yml` on this branch — it predates Antora adoption | +| `unparsed` | No UI bundle release-download URL matched the expected shape in the playbook — needs a look | +| `error` | An API call failed; the detail is in the summary | + +## Rerunning + +Safe to run repeatedly, for the same reason `update-maven-wrapper.yml` is — it derives +everything from current state, never from a queue: + +| On rerun | Result | +|---|---| +| PR still open at the target, checks green | `pr-open` — merged only on a manual run with `auto_merge` on; a scheduled run leaves it | +| PR still open at the target, checks red or pending | `pr-open` — left for a human | +| PR merged | The branch is now current → `up-to-date` | +| PR open, but a **newer release** has since shipped | `pr-updated` — the commit lands on the **existing** PR's branch, moving it up in place | +| PR closed unmerged, branch still present | `branch-exists` — left alone, so a deliberate rejection is not re-litigated | +| PR closed unmerged, branch deleted | A fresh PR is opened | + +**One PR per repo/branch, always at the current target.** The head branch name includes both +the base branch and the target tag — +`antora-ui-bundle-update/-` — and existing PRs are matched by prefix and base +rather than by exact head name, so a PR opened for an earlier release is found and moved up +rather than having a second PR stacked on top of it. + +## Notes + +- One branch failing never stops the others (`fail-fast: false`, `max-parallel: 8`); errors + are collected and listed in the summary. +- A `Release Freeze` from [`lock-unlock-branches.yml`](README-lock-branches.md) restricts + pushes to the frozen release branches, so the PR branch itself is created fine, but the PR + cannot be merged until the freeze lifts. diff --git a/.github/workflows/ci-status-report.yml b/.github/workflows/ci-status-report.yml index fa883f7..6d3d3ce 100644 --- a/.github/workflows/ci-status-report.yml +++ b/.github/workflows/ci-status-report.yml @@ -100,41 +100,25 @@ jobs: env: PROJECTS_FILTER: ${{ inputs.projects }} REPO_TYPE: ${{ inputs.repo_type }} + MATRIX_LIB: ${{ github.workspace }}/.github/scripts/project-branch-matrix.js run: | node - << 'JSEOF' const fs = require('fs'); + const { buildBranchMatrix } = require(process.env.MATRIX_LIB); const projects = JSON.parse(fs.readFileSync('config/projects.json', 'utf8')); - const filterRaw = (process.env.PROJECTS_FILTER || '').trim(); - const filter = filterRaw - ? new Set(filterRaw.split(',').map(p => p.trim()).filter(Boolean)) - : new Set(); - - const repoType = (process.env.REPO_TYPE || 'both').trim(); - const typeKeys = repoType === 'both' ? ['oss', 'commercial'] : [repoType]; - - // Expand every project's oss/commercial section into one matrix entry - // per branch it lists as "scheduled" - the full set of maintained - // branches, as opposed to "default" which is just the primary one. - const entries = []; - for (const [projectKey, config] of Object.entries(projects)) { - if (projectKey === 'defaults') continue; - if (filter.size > 0 && !filter.has(projectKey)) continue; - for (const typeKey of typeKeys) { - if (!config[typeKey]) continue; - const branches = config[typeKey]?.branches?.scheduled || []; - const repo = typeKey === 'commercial' - ? `spring-cloud/${projectKey}-commercial` - : `spring-cloud/${projectKey}`; - for (const branch of branches) { - entries.push({ project: projectKey, repo, type: typeKey, branch }); - } - } - } - - entries.sort((a, b) => - a.repo.localeCompare(b.repo) || a.branch.localeCompare(b.branch)); + // Expand every project's oss/commercial section into one matrix entry per branch + // it lists as "scheduled" - the full set of maintained branches, as opposed to + // "default" which is just the primary one. `-internal` branches are kept + // (excludeInternal: false): they run ci-release.yml just like some OSS release/* + // branches do, and this workflow checks whichever of ci-release.yml/ci.yml/ci.yaml + // actually exists on the branch. + const { entries } = buildBranchMatrix(projects, { + filter: process.env.PROJECTS_FILTER, + repoType: process.env.REPO_TYPE, + excludeInternal: false, + }); console.log(`Branches to check: ${entries.length}`); for (const e of entries) console.log(` ${e.repo}@${e.branch} (${e.type})`); diff --git a/.github/workflows/dependabot-report.yml b/.github/workflows/dependabot-report.yml index 29bfeea..d2e2654 100644 --- a/.github/workflows/dependabot-report.yml +++ b/.github/workflows/dependabot-report.yml @@ -74,45 +74,23 @@ jobs: env: PROJECTS_FILTER: ${{ inputs.projects }} REPO_TYPE: ${{ inputs.repo_type }} + MATRIX_LIB: ${{ github.workspace }}/.github/scripts/project-branch-matrix.js run: | node - << 'JSEOF' const fs = require('fs'); + const { buildRepoMatrix } = require(process.env.MATRIX_LIB); const projects = JSON.parse(fs.readFileSync('config/projects.json', 'utf8')); - const filterRaw = (process.env.PROJECTS_FILTER || '').trim(); - const filter = filterRaw - ? new Set(filterRaw.split(',').map(p => p.trim()).filter(Boolean)) - : new Set(); - - const repoType = (process.env.REPO_TYPE || 'both').trim(); - const typeKeys = repoType === 'both' ? ['oss', 'commercial'] : [repoType]; - // One entry per repository, not per branch - Dependabot PRs are listed // repo-wide and then attributed to a branch, so fanning out per branch would - // fetch the same PR list several times. - const entries = []; - for (const [projectKey, config] of Object.entries(projects)) { - if (projectKey === 'defaults') continue; - if (filter.size > 0 && !filter.has(projectKey)) continue; - for (const typeKey of typeKeys) { - if (!config[typeKey]) continue; - const branches = config[typeKey]?.branches?.scheduled || []; - const repo = typeKey === 'commercial' - ? `spring-cloud/${projectKey}-commercial` - : `spring-cloud/${projectKey}`; - entries.push({ - project: projectKey, - repo, - type: typeKey, - // Comma-separated: a matrix cannot carry an array through an expression - // without toJson pretty-printing it and breaking the consuming YAML. - branches: branches.join(','), - }); - } - } - - entries.sort((a, b) => a.repo.localeCompare(b.repo)); + // fetch the same PR list several times. Branches are comma-joined because a + // matrix cannot carry an array through an expression without `toJson` + // pretty-printing it and breaking the consuming YAML. + const { entries } = buildRepoMatrix(projects, { + filter: process.env.PROJECTS_FILTER, + repoType: process.env.REPO_TYPE, + }); console.log(`Repositories to scan: ${entries.length}`); for (const e of entries) console.log(` ${e.repo} (${e.type}) [${e.branches}]`); diff --git a/.github/workflows/dependabot-triage.yml b/.github/workflows/dependabot-triage.yml index 0304760..d357ee9 100644 --- a/.github/workflows/dependabot-triage.yml +++ b/.github/workflows/dependabot-triage.yml @@ -96,42 +96,20 @@ jobs: env: PROJECTS_FILTER: ${{ inputs.projects }} REPO_TYPE: ${{ inputs.repo_type || 'both' }} + MATRIX_LIB: ${{ github.workspace }}/.github/scripts/project-branch-matrix.js run: | node - << 'JSEOF' const fs = require('fs'); + const { buildRepoMatrix } = require(process.env.MATRIX_LIB); const projects = JSON.parse(fs.readFileSync('config/projects.json', 'utf8')); - const filterRaw = (process.env.PROJECTS_FILTER || '').trim(); - const filter = filterRaw - ? new Set(filterRaw.split(',').map(p => p.trim()).filter(Boolean)) - : new Set(); - - const repoType = (process.env.REPO_TYPE || 'both').trim(); - const typeKeys = repoType === 'both' ? ['oss', 'commercial'] : [repoType]; - // One entry per repository, matching dependabot-report.yml - Dependabot PRs are // listed repo-wide and then attributed to a branch. - const entries = []; - for (const [projectKey, config] of Object.entries(projects)) { - if (projectKey === 'defaults') continue; - if (filter.size > 0 && !filter.has(projectKey)) continue; - for (const typeKey of typeKeys) { - if (!config[typeKey]) continue; - const branches = config[typeKey]?.branches?.scheduled || []; - const repo = typeKey === 'commercial' - ? `spring-cloud/${projectKey}-commercial` - : `spring-cloud/${projectKey}`; - entries.push({ - project: projectKey, - repo, - type: typeKey, - branches: branches.join(','), - }); - } - } - - entries.sort((a, b) => a.repo.localeCompare(b.repo)); + const { entries } = buildRepoMatrix(projects, { + filter: process.env.PROJECTS_FILTER, + repoType: process.env.REPO_TYPE, + }); console.log(`Repositories to triage: ${entries.length}`); for (const e of entries) console.log(` ${e.repo} (${e.type}) [${e.branches}]`); diff --git a/.github/workflows/lock-unlock-branches.yml b/.github/workflows/lock-unlock-branches.yml index a7a9992..98ea4fc 100644 --- a/.github/workflows/lock-unlock-branches.yml +++ b/.github/workflows/lock-unlock-branches.yml @@ -125,17 +125,16 @@ jobs: PROJECTS_FILTER: ${{ inputs.projects }} REPO_TYPE: ${{ inputs.repo_type }} UNLOCK: ${{ inputs.unlock }} + MATRIX_LIB: ${{ github.workspace }}/.github/scripts/project-branch-matrix.js run: | node - << 'JSEOF' const fs = require('fs'); + const { parseProjectFilter, knownProjectKeys, buildRepoMatrix } = + require(process.env.MATRIX_LIB); const projects = JSON.parse(fs.readFileSync('config/projects.json', 'utf8')); - const filterRaw = (process.env.PROJECTS_FILTER || '').trim(); - const filter = filterRaw - ? new Set(filterRaw.split(',').map(p => p.trim()).filter(Boolean)) - : new Set(); - + const filter = parseProjectFilter(process.env.PROJECTS_FILTER); const repoType = (process.env.REPO_TYPE || 'both').trim(); const unlock = (process.env.UNLOCK || 'false') === 'true'; @@ -147,12 +146,11 @@ jobs: if (!['both', 'oss', 'commercial'].includes(repoType)) { fail(`ERROR: repo_type must be one of both, oss, commercial; got '${repoType}'.`); } - const typeKeys = repoType === 'both' ? ['oss', 'commercial'] : [repoType]; // A typo in projects would otherwise process nothing and report success, and "no rows" // reads as "nothing needed doing" - the last thing you want to believe about a freeze. if (filter.size) { - const known = Object.keys(projects).filter(k => k !== 'defaults'); + const known = knownProjectKeys(projects); const unknown = [...filter].filter(p => !known.includes(p)); if (unknown.length) { fail(`ERROR: not found in projects.json: ${unknown.join(', ')}`, @@ -163,38 +161,25 @@ jobs: // One entry per repository, not per branch. All of a repository's frozen branches live // in one ruleset, so a per-branch fan-out would have several legs writing the same - // object at once. The branch list rides along as a comma-separated string because a - // matrix cannot carry an array without toJson pretty-printing it and breaking the - // consuming YAML - the same reason dependabot-report.yml joins its branches. - const entries = []; - let branchCount = 0; - for (const [projectKey, config] of Object.entries(projects)) { - if (projectKey === 'defaults') continue; - if (filter.size > 0 && !filter.has(projectKey)) continue; - for (const typeKey of typeKeys) { - if (!config[typeKey]) continue; - - // Everything under branches, not just scheduled: scheduled is the maintained set - // and default is the primary branch, and a freeze that let commits through on the - // default branch would be worse than no freeze at all. default is a subset of - // scheduled throughout projects.json today, but nothing enforces that. - const branches = config[typeKey].branches || {}; - const all = [...new Set([...(branches.scheduled || []), ...(branches.default || [])])]; - if (!all.length) continue; - - entries.push({ - project: projectKey, - repo: typeKey === 'commercial' - ? `spring-cloud/${projectKey}-commercial` - : `spring-cloud/${projectKey}`, - type: typeKey, - branches: all.join(','), - }); - branchCount += all.length; - } - } - - entries.sort((a, b) => a.repo.localeCompare(b.repo)); + // object at once. Everything under branches, not just scheduled: scheduled is the + // maintained set and default is the primary branch, and a freeze that let commits + // through on the default branch would be worse than no freeze at all. default is a + // subset of scheduled throughout projects.json today, but nothing enforces that. The + // branch list rides along as a comma-separated string because a matrix cannot carry + // an array without toJson pretty-printing it and breaking the consuming YAML - the + // same reason dependabot-report.yml joins its branches. + const { entries: all } = buildRepoMatrix(projects, { + filter, repoType, + branchesOf: section => { + const branches = section.branches || {}; + return [...new Set([...(branches.scheduled || []), ...(branches.default || [])])]; + }, + }); + // A repository with nothing to freeze is dropped rather than kept as an + // empty-branches row, which a per-branch fan-out would apply the ruleset to nothing + // for - indistinguishable from success in the log. + const entries = all.filter(e => e.branches.length > 0); + const branchCount = entries.reduce((n, e) => n + e.branches.split(',').length, 0); console.log(`Repositories to ${unlock ? 'unlock' : 'lock'}: ${entries.length} ` + `(${branchCount} branches)`); diff --git a/.github/workflows/rollout-actions-ref.yml b/.github/workflows/rollout-actions-ref.yml index e2bcdda..b1d537d 100644 --- a/.github/workflows/rollout-actions-ref.yml +++ b/.github/workflows/rollout-actions-ref.yml @@ -107,53 +107,38 @@ jobs: env: PROJECTS_FILTER: ${{ inputs.projects }} REPO_TYPE: ${{ inputs.repo_type }} + MATRIX_LIB: ${{ github.workspace }}/.github/scripts/project-branch-matrix.js run: | node - << 'JSEOF' const fs = require('fs'); + const { buildBranchMatrix } = require(process.env.MATRIX_LIB); const projects = JSON.parse(fs.readFileSync('config/projects.json', 'utf8')); - const filterRaw = (process.env.PROJECTS_FILTER || '').trim(); - const filter = filterRaw - ? new Set(filterRaw.split(',').map(p => p.trim()).filter(Boolean)) - : new Set(); - - const repoType = (process.env.REPO_TYPE || 'both').trim(); - const typeKeys = repoType === 'both' ? ['oss', 'commercial'] : [repoType]; - // One matrix entry per (repo, branch). branches.scheduled is the set of // branches actively built, which is exactly the set worth migrating - // retired branches are removed from it by the retire-branch workflow. - const include = []; - for (const [projectKey, config] of Object.entries(projects)) { - if (projectKey === 'defaults') continue; - if (filter.size > 0 && !filter.has(projectKey)) continue; - for (const typeKey of typeKeys) { - const section = config[typeKey]; - if (!section) continue; - const repo = typeKey === 'commercial' - ? `spring-cloud/${projectKey}-commercial` - : `spring-cloud/${projectKey}`; - for (const branch of section.branches?.scheduled || []) { - // -internal branches are rebased from their OSS branch, so ci.yml - // and pr.yml there are overwritten from OSS and must not be pinned - // here. ci-release.yml is internal-only and survives the rebase, so - // it is the one file the rollout owns on those branches - and the - // one file it must not touch anywhere else. - const internal = branch.endsWith('-internal'); - include.push({ - repo, - branch, - type: typeKey, - scope: internal ? 'ci-release.yml only' : 'all but ci-release.yml', - includeFiles: internal ? 'ci-release.yml' : '', - excludeFiles: internal ? '' : 'ci-release.yml', - }); - } - } - } - - include.sort((a, b) => a.repo.localeCompare(b.repo) || a.branch.localeCompare(b.branch)); + // `-internal` branches are kept (excludeInternal: false) and given their own + // scope below rather than excluded: they are rebased from their OSS branch, so + // ci.yml and pr.yml there are overwritten from OSS and must not be pinned here. + // ci-release.yml is internal-only and survives the rebase, so it is the one file + // the rollout owns on those branches - and the one file it must not touch + // anywhere else. + const { entries } = buildBranchMatrix(projects, { + filter: process.env.PROJECTS_FILTER, + repoType: process.env.REPO_TYPE, + excludeInternal: false, + }); + + const include = entries.map(({ repo, branch, type }) => { + const internal = branch.endsWith('-internal'); + return { + repo, branch, type, + scope: internal ? 'ci-release.yml only' : 'all but ci-release.yml', + includeFiles: internal ? 'ci-release.yml' : '', + excludeFiles: internal ? '' : 'ci-release.yml', + }; + }); console.log(`${include.length} (repo, branch) pairs:`); for (const e of include) console.log(` ${e.repo}@${e.branch} [${e.scope}]`); diff --git a/.github/workflows/test-maven-wrapper-properties.yml b/.github/workflows/test-maven-wrapper-properties.yml deleted file mode 100644 index 812d1c5..0000000 --- a/.github/workflows/test-maven-wrapper-properties.yml +++ /dev/null @@ -1,100 +0,0 @@ -name: Test - Maven Wrapper Properties - -# Unit tests for .github/scripts/maven-wrapper-properties.js, the shared rules both halves -# of update-maven-wrapper.yml edit maven-wrapper.properties by. -# -# There is deliberately no "dist is up to date" job here, unlike the tests for the actions -# under .github/actions. Those are node20 actions with npm dependencies: they run from a -# consumer's checkout with no install step, so ncc has to bundle them into dist/index.js. -# This file is neither - it has no dependencies and no action.yml, and is required directly -# by the inline node scripts in the workflow after this repo is checked out. Bundling it -# would produce an identical file and one more way for CI to fail. - -on: - push: - paths: - - '.github/scripts/**' - - '.github/workflows/update-maven-wrapper.yml' - - '.github/workflows/test-maven-wrapper-properties.yml' - pull_request: - paths: - - '.github/scripts/**' - - '.github/workflows/update-maven-wrapper.yml' - - '.github/workflows/test-maven-wrapper-properties.yml' - -jobs: - unit-tests: - name: Unit Tests - runs-on: ubuntu-latest - defaults: - run: - working-directory: .github/scripts - steps: - - uses: actions/checkout@v4 - - - uses: actions/setup-node@v4 - with: - node-version: '20' - cache: 'npm' - cache-dependency-path: .github/scripts/package.json - - - name: Install dependencies - run: npm install - - - name: Run unit tests - run: npm test - - # ── The workflow's own inline scripts ─────────────────────────────────────── - # The scripts that use this library live inside YAML heredocs, where a syntax error is - # invisible until the workflow runs against a real repository. Extracting and parsing - # them here turns that into a pull-request failure instead. - inline-scripts: - name: Workflow inline scripts parse - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - - - uses: actions/setup-node@v4 - with: - node-version: '20' - - - name: Extract and syntax-check every inline node block - run: | - set -euo pipefail - WF=.github/workflows/update-maven-wrapper.yml - OUT=$(mktemp -d) - awk -v out="$OUT" ' - /^ +node - << .JSEOF.$/ { n++; f = out "/block-" n ".js"; inblock = 1; next } - inblock && /^ +JSEOF$/ { inblock = 0; next } - inblock { sub(/^ /, ""); print > f } - ' "$WF" - - count=$(ls "$OUT" | wc -l | tr -d ' ') - echo "Extracted ${count} inline node block(s) from ${WF}" - - # Guards against the extraction silently matching nothing if the heredoc marker - # or indentation is ever changed - which would turn this job into a no-op that - # always passes. - if [ "${count}" -lt 5 ]; then - echo "❌ expected at least 5 blocks; the heredoc markers may have changed" - exit 1 - fi - - for f in "${OUT}"/*.js; do - node --check "$f" - echo " ✅ $(basename "$f")" - done - - - name: Verify the library loads and exports what the workflow uses - run: | - node -e ' - const W = require("./.github/scripts/maven-wrapper-properties.js"); - const used = ["wrapperDirs", "propsPath", "scriptPaths", "currentMaven", - "dependabotWrapperVersion", "rewrite", "cmp"]; - const missing = used.filter(n => typeof W[n] !== "function"); - if (missing.length) { - console.log(`❌ missing exports the workflow calls: ${missing.join(", ")}`); - process.exit(1); - } - console.log(`✅ all ${used.length} exports the workflow calls are present`); - ' diff --git a/.github/workflows/test-shared-scripts.yml b/.github/workflows/test-shared-scripts.yml new file mode 100644 index 0000000..d17a660 --- /dev/null +++ b/.github/workflows/test-shared-scripts.yml @@ -0,0 +1,163 @@ +name: Test - Shared Scripts + +# Unit tests for the .github/scripts/*.js modules shared across the workflows below - the +# config/projects.json matrix walk (project-branch-matrix.js), the version comparator +# (version-cmp.js), the gh-CLI and git-data-API/PR mechanics (gh-cli.js, git-pr-helpers.js), +# the green-PR merge check (merge-if-green.js), and the two domain-specific modules that sit +# on top of them (maven-wrapper-properties.js, antora-ui-bundle.js). +# +# There is deliberately no "dist is up to date" job here, unlike the tests for the actions +# under .github/actions. These are node20 actions with npm dependencies: they run from a +# consumer's checkout with no install step, so ncc has to bundle them into dist/index.js. +# The modules here are neither - they have no dependencies and no action.yml, and are +# required directly by the inline node scripts in each workflow after this repo is checked +# out. Bundling them would produce an identical file and one more way for CI to fail. + +on: + push: + paths: + - '.github/scripts/**' + - '.github/workflows/update-maven-wrapper.yml' + - '.github/workflows/update-antora-ui-bundle.yml' + - '.github/workflows/rollout-actions-ref.yml' + - '.github/workflows/ci-status-report.yml' + - '.github/workflows/dependabot-report.yml' + - '.github/workflows/dependabot-triage.yml' + - '.github/workflows/lock-unlock-branches.yml' + - '.github/workflows/test-shared-scripts.yml' + pull_request: + paths: + - '.github/scripts/**' + - '.github/workflows/update-maven-wrapper.yml' + - '.github/workflows/update-antora-ui-bundle.yml' + - '.github/workflows/rollout-actions-ref.yml' + - '.github/workflows/ci-status-report.yml' + - '.github/workflows/dependabot-report.yml' + - '.github/workflows/dependabot-triage.yml' + - '.github/workflows/lock-unlock-branches.yml' + - '.github/workflows/test-shared-scripts.yml' + +jobs: + unit-tests: + name: Unit Tests + runs-on: ubuntu-latest + defaults: + run: + working-directory: .github/scripts + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-node@v4 + with: + node-version: '20' + cache: 'npm' + cache-dependency-path: .github/scripts/package.json + + - name: Install dependencies + run: npm install + + - name: Run unit tests + run: npm test + + # ── Every consuming workflow's own inline scripts ─────────────────────────────── + # The scripts that use these modules live inside YAML heredocs, where a syntax error is + # invisible until the workflow runs against a real repository. Extracting and parsing them + # here turns that into a pull-request failure instead. One matrix leg per consuming + # workflow rather than one job per workflow: the extraction/check steps are identical, only + # the block count and the exports each workflow actually calls differ. + inline-scripts: + name: "Inline scripts — ${{ matrix.workflow }}" + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + include: + - workflow: update-maven-wrapper.yml + min-blocks: 5 + exports: >- + gh-cli.js:ghRetry| + git-pr-helpers.js:readFileAt,createBranch,commitFilesToRef,findOpenPrByHeadPrefix,openPr| + merge-if-green.js:mergeIfGreen| + project-branch-matrix.js:buildBranchMatrix| + version-cmp.js:cmp| + maven-wrapper-properties.js:wrapperDirs,propsPath,scriptPaths,currentMaven,dependabotWrapperVersion,rewrite,cmp + - workflow: update-antora-ui-bundle.yml + min-blocks: 4 + exports: >- + git-pr-helpers.js:readFileAt,createBranch,commitFilesToRef,findOpenPrByHeadPrefix,openPr| + merge-if-green.js:mergeIfGreen| + project-branch-matrix.js:buildBranchMatrix| + antora-ui-bundle.js:PLAYBOOK_PATH,extractBundle,rewrite,cmp + - workflow: rollout-actions-ref.yml + min-blocks: 2 + exports: 'project-branch-matrix.js:buildBranchMatrix' + - workflow: ci-status-report.yml + min-blocks: 2 + exports: 'project-branch-matrix.js:buildBranchMatrix' + - workflow: dependabot-report.yml + min-blocks: 2 + exports: 'project-branch-matrix.js:buildRepoMatrix' + - workflow: dependabot-triage.yml + min-blocks: 3 + exports: 'project-branch-matrix.js:buildRepoMatrix' + - workflow: lock-unlock-branches.yml + min-blocks: 3 + exports: 'project-branch-matrix.js:parseProjectFilter,knownProjectKeys,buildRepoMatrix' + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-node@v4 + with: + node-version: '20' + + - name: Extract and syntax-check every inline node block + env: + WORKFLOW: ${{ matrix.workflow }} + MIN_BLOCKS: ${{ matrix['min-blocks'] }} + run: | + set -euo pipefail + WF=".github/workflows/${WORKFLOW}" + OUT=$(mktemp -d) + awk -v out="$OUT" ' + /^ +node - << .JSEOF.$/ { n++; f = out "/block-" n ".js"; inblock = 1; next } + inblock && /^ +JSEOF$/ { inblock = 0; next } + inblock { sub(/^ /, ""); print > f } + ' "$WF" + + count=$(ls "$OUT" | wc -l | tr -d ' ') + echo "Extracted ${count} inline node block(s) from ${WF}" + + # Guards against the extraction silently matching nothing if the heredoc marker + # or indentation is ever changed - which would turn this job into a no-op that + # always passes. + if [ "${count}" -lt "${MIN_BLOCKS}" ]; then + echo "❌ expected at least ${MIN_BLOCKS} blocks; the heredoc markers may have changed" + exit 1 + fi + + for f in "${OUT}"/*.js; do + node --check "$f" + echo " ✅ $(basename "$f")" + done + + - name: Verify the shared modules load and export what this workflow uses + env: + EXPORTS: ${{ matrix.exports }} + run: | + node -e ' + const path = require("path"); + const specs = process.env.EXPORTS.split("|").map(s => s.trim()).filter(Boolean); + const missing = []; + for (const spec of specs) { + const [file, namesCsv] = spec.split(":"); + const mod = require(path.resolve(".github/scripts", file)); + for (const name of namesCsv.split(",")) { + if (typeof mod[name] === "undefined") missing.push(`${file}:${name}`); + } + } + if (missing.length) { + console.log(`❌ missing exports: ${missing.join(", ")}`); + process.exit(1); + } + console.log(`✅ all exports present: ${specs.join(", ")}`); + ' diff --git a/.github/workflows/update-antora-ui-bundle.yml b/.github/workflows/update-antora-ui-bundle.yml new file mode 100644 index 0000000..9afcce4 --- /dev/null +++ b/.github/workflows/update-antora-ui-bundle.yml @@ -0,0 +1,528 @@ +name: Update Antora UI Bundle + +run-name: "${{ (github.event_name != 'schedule' && inputs.dry_run != false) && 'Check' || 'Update' }} Antora UI Bundle${{ inputs.projects != '' && format(' [{0}]', inputs.projects) || '' }} (${{ inputs.repo_type || 'both' }})" + +# Opens a pull request against every maintained repository/branch whose docs/antora-playbook.yml +# points at an old release of the Antora UI bundle (spring-io/antora-ui-spring), so CI decides +# whether the bump is safe rather than nobody noticing the docs look stale. +# +# 1. versions - resolve the latest spring-io/antora-ui-spring release tag +# 2. setup - read config/projects.json and build one matrix entry per repo/branch +# 3. update - per repo/branch: compare, and open a PR if it is behind +# 4. summary - one report covering every combination, posted to Google Chat when +# anything changed +# +# docs/antora-playbook.yml is a per-branch file - content.sources[].branches is HEAD, so it +# documents its own branch rather than there being one canonical copy on main. Branches that +# predate Antora adoption have no such file at all, which this discovers rather than assumes +# (confirmed: spring-cloud-commons's 3.1.x has none, while main/4.2.x/4.3.x/5.0.x do). +# +# This is the same shape of problem update-maven-wrapper.yml solves for the Maven wrapper - +# an upstream artifact versions independently of every project, and nothing else notices when +# it moves - and shares that workflow's gh-cli.js, git-pr-helpers.js, merge-if-green.js and +# project-branch-matrix.js rather than reimplementing the same mechanics. +# +# See README-update-antora-ui-bundle.md for details. + +on: + workflow_dispatch: + inputs: + projects: + description: 'Comma-separated list of Spring Cloud project names. When empty, every project in projects.json is processed.' + required: false + type: string + default: '' + repo_type: + description: 'Update commercial, oss, or both?' + required: false + type: choice + default: 'both' + options: + - both + - oss + - commercial + bundle_version: + description: 'Target spring-io/antora-ui-spring release tag (e.g. v0.4.26). When empty, the latest release is used.' + required: false + type: string + default: '' + auto_merge: + description: 'Merge existing UI-bundle PRs whose checks have all passed. Applies to this manual run only - the weekly scheduled run never merges.' + required: false + type: boolean + default: true + merge_method: + description: 'How to merge a green UI-bundle PR.' + required: false + type: choice + default: 'squash' + options: + - squash + - merge + - rebase + dry_run: + description: 'Dry run, if checked nothing is created, updated or merged, but you can see what would change' + required: false + type: boolean + default: true + notify: + description: 'Post a Google Chat notification when anything changed. Always on for the scheduled run.' + required: false + type: boolean + default: true + token: + description: 'GitHub token with contents:write and pull-requests:write on all target repos. Falls back to GH_ACTIONS_REPO_TOKEN.' + required: false + type: string + default: '' + + # Mondays at ~10:00am US Eastern - a few hours after update-maven-wrapper.yml's ~7:00am + # schedule, so the two weekly runs don't land on the shared runner pool at the same moment. + # GitHub Actions cron is always UTC with no notion of DST, hence the two month-selected + # entries (EDT/EST). + schedule: + - cron: '0 14 * 3-10 1' # ~10:00am EDT Monday, March-October + - cron: '0 15 * 11,12,1,2 1' # ~10:00am EST Monday, November-February + +permissions: + contents: read + +jobs: + versions: + name: Resolve Target Version + runs-on: ubuntu-latest + outputs: + tag: ${{ steps.resolve.outputs.tag }} + steps: + - name: Resolve latest UI bundle release + id: resolve + env: + GH_TOKEN: ${{ inputs.token || secrets.GH_ACTIONS_REPO_TOKEN }} + BUNDLE_VERSION_INPUT: ${{ inputs.bundle_version }} + run: | + set -euo pipefail + TAG="${BUNDLE_VERSION_INPUT:-}" + if [[ -z "$TAG" ]]; then + TAG=$(gh api repos/spring-io/antora-ui-spring/releases/latest --jq .tag_name) + fi + if [[ -z "$TAG" ]]; then + echo "::error::Could not resolve the latest spring-io/antora-ui-spring release." + exit 1 + fi + echo "Target UI bundle release: ${TAG}" + echo "tag=${TAG}" >> "$GITHUB_OUTPUT" + + setup: + name: Build Matrix + runs-on: ubuntu-latest + outputs: + matrix: ${{ steps.build-matrix.outputs.matrix }} + count: ${{ steps.build-matrix.outputs.count }} + excluded: ${{ steps.build-matrix.outputs.excluded }} + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Build matrix + id: build-matrix + env: + PROJECTS_FILTER: ${{ inputs.projects }} + REPO_TYPE: ${{ inputs.repo_type || 'both' }} + MATRIX_LIB: ${{ github.workspace }}/.github/scripts/project-branch-matrix.js + run: | + node - << 'JSEOF' + const fs = require('fs'); + const { buildBranchMatrix } = require(process.env.MATRIX_LIB); + + const projects = JSON.parse(fs.readFileSync('config/projects.json', 'utf8')); + + // One entry per branch: docs/antora-playbook.yml is per-branch (it documents its + // own branch via content.sources[].branches: HEAD), so every maintained branch + // carries its own copy. `-internal` branches are excluded by default - same + // reasoning as update-maven-wrapper.yml's setup job: they are the in-flight + // development line for the next train, so an unsolicited change there lands in + // the middle of active work rather than on a settled branch. + const { entries, excluded } = buildBranchMatrix(projects, { + filter: process.env.PROJECTS_FILTER, + repoType: process.env.REPO_TYPE, + }); + + console.log(`Repo/branch combinations to check: ${entries.length}`); + for (const e of entries) console.log(` ${e.repo}@${e.branch}`); + console.log(`Excluded (-internal): ${excluded.length}`); + for (const e of excluded) console.log(` ${e}`); + + fs.appendFileSync(process.env.GITHUB_OUTPUT, + `matrix=${JSON.stringify({ include: entries })}\n`); + fs.appendFileSync(process.env.GITHUB_OUTPUT, `count=${entries.length}\n`); + fs.appendFileSync(process.env.GITHUB_OUTPUT, + `excluded=${JSON.stringify(excluded)}\n`); + JSEOF + + update: + name: "Antora UI Bundle — ${{ matrix.repo }}@${{ matrix.branch }}" + needs: [versions, setup] + if: needs.setup.outputs.count != '0' + runs-on: ubuntu-latest + strategy: + fail-fast: false + max-parallel: 8 + matrix: ${{ fromJson(needs.setup.outputs.matrix) }} + steps: + # This repository, for the shared .github/scripts/*.js modules - antora-ui-bundle.js + # (the rules this step edits docs/antora-playbook.yml by) and git-pr-helpers.js (the + # generic git-data-API/PR mechanics, shared with update-maven-wrapper.yml). Checked out + # into a subdirectory on purpose: a checkout at the workspace root would clean the + # antora-ui-*.json result file this step writes there. + - name: Checkout shared scripts + uses: actions/checkout@v4 + with: + path: ci-actions + + - name: Check and update UI bundle + id: update + env: + GH_TOKEN: ${{ inputs.token || secrets.GH_ACTIONS_REPO_TOKEN }} + BUNDLE_LIB: ${{ github.workspace }}/ci-actions/.github/scripts/antora-ui-bundle.js + GIT_PR_LIB: ${{ github.workspace }}/ci-actions/.github/scripts/git-pr-helpers.js + REPO: ${{ matrix.repo }} + BRANCH: ${{ matrix.branch }} + PROJECT: ${{ matrix.project }} + TYPE: ${{ matrix.type }} + TARGET_TAG: ${{ needs.versions.outputs.tag }} + # A scheduled run has no inputs, so `inputs.dry_run != false` would be true and the + # weekly job would never actually do anything. Schedule therefore acts, while a + # manual dispatch still defaults to a dry run so a human can look first - same + # convention as update-maven-wrapper.yml. + DRY_RUN: ${{ github.event_name == 'schedule' && 'false' || (inputs.dry_run != false) }} + run: | + node - << 'JSEOF' + const fs = require('fs'); + + const PR = require(process.env.GIT_PR_LIB); + const A = require(process.env.BUNDLE_LIB); + + const REPO = process.env.REPO; + const BRANCH = process.env.BRANCH; + const TARGET = process.env.TARGET_TAG; + const DRY = process.env.DRY_RUN !== 'false'; + + const safe = `${REPO}@${BRANCH}`.replace(/[/@]/g, '-'); + const finish = (status, detail, extra = {}) => { + const result = { project: process.env.PROJECT, repo: REPO, type: process.env.TYPE, + branch: BRANCH, status, detail, dryRun: DRY, ...extra }; + fs.writeFileSync(`antora-ui-${safe}.json`, JSON.stringify(result, null, 2)); + fs.appendFileSync(process.env.GITHUB_OUTPUT, `safe-name=${safe}\n`); + // Only a PR that is already open AND already at the target is a merge + // candidate. One just opened, or just moved up, has no finished CI yet. + if (status === 'pr-open' && extra.prNumber) { + fs.appendFileSync(process.env.GITHUB_OUTPUT, `merge-candidate=${extra.prNumber}\n`); + } + console.log(`${REPO}@${BRANCH}: ${status}${detail ? ` - ${detail}` : ''}`); + process.exit(0); + }; + + // Encoded for query strings; left raw inside a path, where GitHub accepts the + // slashes a branch name may contain. + const enc = encodeURIComponent(BRANCH); + + // ── Read the playbook at this branch ───────────────────────────────────────────── + const got = PR.readFileAt(REPO, A.PLAYBOOK_PATH, enc); + if (!got) finish('no-playbook', `no ${A.PLAYBOOK_PATH} on this branch`); + + const bundle = A.extractBundle(got.text); + if (!bundle) finish('unparsed', `no UI bundle release-download url in ${A.PLAYBOOK_PATH}`); + + const extras = { current: bundle.tag, target: TARGET }; + + // Never walk a file backwards - a manual override, or a target resolved before + // this branch was last touched, both leave the branch pinned ahead of the target. + const cmpResult = A.cmp(bundle.tag, TARGET); + if (cmpResult > 0) finish('ahead', `${bundle.tag} is newer than the target ${TARGET}`, extras); + if (cmpResult === 0) finish('up-to-date', `already on ${bundle.tag}`, extras); + + // ── Is there already a PR for this branch? ────────────────────────────────────── + // The head name includes the target tag, mirroring update-maven-wrapper.yml's + // convention, so a PR opened for an earlier release is found and bumped in place + // rather than stacking a second PR on top of it. + const slug = BRANCH.replace(/\//g, '-'); + const PREFIX = 'antora-ui-bundle-update/'; + const head = `${PREFIX}${slug}-${TARGET}`; + + const authorName = 'spring-builds'; + const authorEmail = 'spring-builds@users.noreply.github.com'; + // Spring Cloud repositories run a required DCO check, so the commit needs a + // Signed-off-by trailer whose address matches the commit author. + const message = `Update Antora UI bundle to ${TARGET}\n\n` + + `Signed-off-by: ${authorName} <${authorEmail}>`; + const author = { name: authorName, email: authorEmail }; + + const prBody = fromTag => [ + `Updates the Antora UI bundle on \`${BRANCH}\` from **${fromTag}** to **${TARGET}**.`, + '', + `Release notes: https://github.com/${bundle.repo}/releases/tag/${TARGET}`, + '', + 'Opened automatically by [update-antora-ui-bundle.yml](https://github.com/spring-cloud/spring-cloud-github-actions/blob/main/.github/workflows/update-antora-ui-bundle.yml).', + ].join('\n'); + + const findPr = PR.findOpenPrByHeadPrefix(REPO, BRANCH, `${PREFIX}${slug}-`); + if (!findPr.ok) finish('error', `could not list pull requests: ${findPr.err}`); + + if (findPr.pr) { + const pr = findPr.pr; + // Read the PR's own head branch: it may already be at the target, or behind it + // (a newer release has shipped since it was opened). + const headRef = encodeURIComponent(pr.head.ref); + const onHead = PR.readFileAt(REPO, A.PLAYBOOK_PATH, headRef); + const headBundle = onHead && A.extractBundle(onHead.text); + + if (!headBundle || A.cmp(headBundle.tag, TARGET) >= 0) { + finish('pr-open', `#${pr.number} already open for ${TARGET}`, + { ...extras, prUrl: pr.html_url, prNumber: pr.number }); + } + if (DRY) { + finish('would-update-pr', `#${pr.number} would be moved up to ${TARGET}`, + { ...extras, prUrl: pr.html_url }); + } + // Commit onto the existing PR's branch so it updates in place - one UI-bundle + // PR per branch, always at the current target, rather than a growing stack. + const headUpdated = A.rewrite(onHead.text, { repo: headBundle.repo, tag: TARGET }); + const bump = PR.commitFilesToRef(REPO, pr.head.ref, + [{ path: A.PLAYBOOK_PATH, content: headUpdated }], message, author); + if (!bump.ok) finish('error', `could not update #${pr.number}: ${bump.err}`); + finish('pr-updated', `#${pr.number} moved up to ${TARGET}`, + { ...extras, prUrl: pr.html_url, prNumber: pr.number }); + } + + if (DRY) finish('would-open', `${bundle.tag} -> ${TARGET}`, extras); + + // ── Create the branch, commit, open the PR ────────────────────────────────────── + const madeRef = PR.createBranch(REPO, head, BRANCH); + if (!madeRef.ok) finish('error', madeRef.err); + // An existing branch with no open PR means a previous PR was closed unmerged. + // Leaving it alone is deliberate - reopening it would re-litigate that decision. + if (madeRef.alreadyExists) { + finish('branch-exists', `${head} exists but no PR is open - left alone`, extras); + } + + const updatedText = A.rewrite(got.text, { repo: bundle.repo, tag: TARGET }); + const put = PR.commitFilesToRef(REPO, head, + [{ path: A.PLAYBOOK_PATH, content: updatedText }], message, author); + if (!put.ok) finish('error', `could not commit: ${put.err}`); + + const opened = PR.openPr(REPO, { + title: `Update Antora UI bundle to ${TARGET}`, + head, base: BRANCH, body: prBody(bundle.tag), + }); + if (!opened.ok) finish('error', opened.err); + + finish('pr-opened', `${bundle.tag} -> ${TARGET}`, + { ...extras, prUrl: opened.pr.html_url, prNumber: opened.pr.number }); + JSEOF + + # Only runs when the previous step found a PR that is open and already at the target + # release - i.e. one whose CI has had a chance to finish. A PR opened or moved up in + # this same run is deliberately not a candidate: its checks have not started yet. + # + # Auto-merge is switched off for the weekly scheduled run: a human decides when a UI + # refresh lands. A manual dispatch still honours the auto_merge input, so merging green + # PRs in bulk is one deliberate click away - same convention as update-maven-wrapper.yml. + - name: Merge PR if all checks pass + if: >- + always() + && github.event_name != 'schedule' + && inputs.auto_merge != false + && steps.update.outputs.merge-candidate != '' + env: + GH_TOKEN: ${{ inputs.token || secrets.GH_ACTIONS_REPO_TOKEN }} + REPO: ${{ matrix.repo }} + PR_NUMBER: ${{ steps.update.outputs.merge-candidate }} + SAFE_NAME: ${{ steps.update.outputs.safe-name }} + MERGE_METHOD: ${{ inputs.merge_method || 'squash' }} + DRY_RUN: ${{ github.event_name == 'schedule' && 'false' || (inputs.dry_run != false) }} + MERGE_LIB: ${{ github.workspace }}/ci-actions/.github/scripts/merge-if-green.js + run: | + node - << 'JSEOF' + const fs = require('fs'); + const { mergeIfGreen } = require(process.env.MERGE_LIB); + + const REPO = process.env.REPO; + const PR = process.env.PR_NUMBER; + const DRY = process.env.DRY_RUN !== 'false'; + const METHOD = process.env.MERGE_METHOD || 'squash'; + const FILE = `antora-ui-${process.env.SAFE_NAME}.json`; + + const { status, detail } = mergeIfGreen({ repo: REPO, pr: PR, method: METHOD, dryRun: DRY }); + + // Merge outcome is folded back into the result the previous step wrote, so the + // summary reports one row per repo/branch rather than two disjoint views. + let result = {}; + try { result = JSON.parse(fs.readFileSync(FILE, 'utf8')); } catch (_) { /* first write */ } + fs.writeFileSync(FILE, JSON.stringify( + { ...result, mergeStatus: status, mergeDetail: detail }, null, 2)); + console.log(`${REPO} #${PR}: ${status}${detail ? ` - ${detail}` : ''}`); + JSEOF + + # Guarded on safe-name: if the script died before writing its result, the artifact + # name would be a bare "antora-ui-", which GitHub rejects and which would turn a + # reporting gap into a job failure. + - name: Upload result + if: always() && steps.update.outputs.safe-name != '' + uses: actions/upload-artifact@v4 + with: + name: antora-ui-${{ steps.update.outputs.safe-name }} + path: antora-ui-${{ steps.update.outputs.safe-name }}.json + if-no-files-found: ignore + + summary: + name: Summary + needs: [versions, setup, update] + runs-on: ubuntu-latest + if: always() + steps: + - name: Download results + uses: actions/download-artifact@v4 + with: + pattern: antora-ui-* + merge-multiple: true + path: results + + - name: Write summary + id: write-summary + env: + TARGET_TAG: ${{ needs.versions.outputs.tag }} + EVENT_NAME: ${{ github.event_name }} + EXCLUDED: ${{ needs.setup.outputs.excluded }} + run: | + node - << 'JSEOF' + const fs = require('fs'); + + let results = []; + try { + results = fs.readdirSync('results') + .filter(f => f.endsWith('.json')) + .map(f => JSON.parse(fs.readFileSync(`results/${f}`, 'utf8'))) + .sort((a, b) => a.repo.localeCompare(b.repo) || a.branch.localeCompare(b.branch)); + } catch (err) { + console.log('No results to summarize.'); + } + + const dry = results.some(r => r.dryRun); + const of = s => results.filter(r => r.status === s); + const withMerge = s => results.filter(r => r.mergeStatus === s); + + const md = ['## Update Antora UI Bundle', '']; + md.push(`Target release **${process.env.TARGET_TAG}**.`, ''); + if (dry) { + md.push('> **Dry run** — no branches or pull requests were created. ' + + 'Re-run with `dry_run` unchecked to open them.', ''); + } + + md.push(`${results.length} repo/branch combination(s) checked — ` + + `**${of('pr-opened').length}** PR(s) opened, ` + + `**${of('pr-updated').length}** existing PR(s) moved up, ` + + `**${of('would-open').length + of('would-update-pr').length}** would change, ` + + `**${of('pr-open').length}** already current, ` + + `**${of('up-to-date').length}** up to date, ` + + `**${of('error').length}** error(s).`, ''); + + const merged = withMerge('merged'); + const wouldMerge = withMerge('would-merge'); + if (merged.length || wouldMerge.length) { + md.push(`**${merged.length}** PR(s) merged` + + (wouldMerge.length ? `, **${wouldMerge.length}** would be merged` : '') + '.', ''); + } + + // Said plainly, so a weekly run with open green PRs and an empty Merge column + // reads as a deliberate policy rather than as merging having quietly failed. + if (process.env.EVENT_NAME === 'schedule') { + md.push('> Auto-merge is off for scheduled runs — open PRs are left for a human. ' + + 'Run this workflow manually with `auto_merge` checked to merge the green ones.', ''); + } + + // Named rather than merely counted: an omission nobody can see is one nobody can + // question, and these are branches a reader would otherwise assume were covered. + let excluded = []; + try { excluded = JSON.parse(process.env.EXCLUDED || '[]'); } catch (_) { /* absent */ } + if (excluded.length) { + md.push('', `### Skipped — \`-internal\` branches (${excluded.length})`, ''); + md.push('Left alone deliberately: these are the in-flight development line for ' + + 'the next train, so the UI bundle is not changed under active work.', ''); + for (const e of excluded) md.push(`- \`${e}\``); + } + + const interesting = results.filter(r => + !['up-to-date', 'no-playbook'].includes(r.status)); + + // Capped rather than exhaustive: a Google Chat message listing every one of ~79 + // branches would bury the count it opens with. `chatSection`-style limit, + // same convention dependabot-report.yml already uses for its own chat text. + const prLines = []; + if (!interesting.length) { + md.push('', 'Every maintained branch with an Antora playbook is already on the target bundle.'); + } else { + md.push('', '| Repo | Branch | Status | Merge | Detail |', '|---|---|---|---|---|'); + for (const r of interesting) { + const detail = r.prUrl ? `[${r.detail}](${r.prUrl})` : (r.detail || ''); + const merge = r.mergeStatus + ? `${r.mergeStatus}${r.mergeDetail ? ` — ${r.mergeDetail}` : ''}` + : ''; + md.push(`| \`${r.repo}\` | \`${r.branch}\` | ${r.status} | ${merge} | ${detail} |`); + if (r.prUrl) prLines.push(`<${r.prUrl}|${r.repo}@${r.branch}>`); + } + } + + const quiet = of('up-to-date').length + of('no-playbook').length; + if (quiet) { + md.push('', `${of('up-to-date').length} branch(es) already current, ` + + `${of('no-playbook').length} with no Antora playbook — omitted above.`); + } + + fs.appendFileSync(process.env.GITHUB_STEP_SUMMARY, md.join('\n') + '\n'); + console.log(md.join('\n')); + + // Chat text is only produced when there is something worth a notification for - a + // weekly "all clear" only teaches people to ignore the channel, same reasoning + // rollout-actions-ref.yml already documents for its own notification. + const changedCount = of('pr-opened').length + of('pr-updated').length + merged.length; + if (changedCount > 0) { + const chat = [`*Update Antora UI Bundle* — target ${process.env.TARGET_TAG}`, + `${of('pr-opened').length} opened, ${of('pr-updated').length} moved up, ` + + `${merged.length} merged.`]; + const capped = prLines.slice(0, 15); + if (capped.length) chat.push('', ...capped); + if (prLines.length > capped.length) chat.push(`…and ${prLines.length - capped.length} more`); + const text = chat.join('\n'); + const delim = `ghadelim_${Date.now()}`; + fs.appendFileSync(process.env.GITHUB_OUTPUT, `chat-text<<${delim}\n${text}\n${delim}\n`); + } + JSEOF + + - name: Send Google Chat notification + if: always() && (github.event_name == 'schedule' || inputs.notify != false) + env: + WEBHOOK_URL: ${{ secrets.SPRING_CLOUD_CORE_CI_GCHAT_WEBHOOK_URL }} + CHAT_TEXT: ${{ steps.write-summary.outputs.chat-text }} + RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} + run: | + set -euo pipefail + + if [[ -z "${WEBHOOK_URL}" ]]; then + echo "SPRING_CLOUD_CORE_CI_GCHAT_WEBHOOK_URL is not set - skipping Google Chat notification." + exit 0 + fi + + if [[ -z "${CHAT_TEXT:-}" ]]; then + echo "Nothing to report - skipping Google Chat notification." + exit 0 + fi + + TEXT=$(printf '%s\n\n<%s|View full report>' "$CHAT_TEXT" "$RUN_URL") + + jq -n --arg text "$TEXT" '{text: $text}' > chat-message.json + + curl --fail --silent --show-error \ + -X POST \ + -H 'Content-Type: application/json; charset=UTF-8' \ + -d @chat-message.json \ + "${WEBHOOK_URL}" diff --git a/.github/workflows/update-maven-wrapper.yml b/.github/workflows/update-maven-wrapper.yml index 45009c6..a09547f 100644 --- a/.github/workflows/update-maven-wrapper.yml +++ b/.github/workflows/update-maven-wrapper.yml @@ -133,15 +133,22 @@ jobs: maven: ${{ steps.resolve.outputs.maven }} wrapper: ${{ steps.resolve.outputs.wrapper }} steps: + # For .github/scripts/version-cmp.js, the dotted-numeric comparator shared with + # maven-wrapper-properties.js and update-antora-ui-bundle.yml. + - name: Checkout + uses: actions/checkout@v4 + - name: Resolve versions id: resolve env: MAVEN_INPUT: ${{ inputs.maven_version }} WRAPPER_INPUT: ${{ inputs.wrapper_version }} + VERSION_CMP_LIB: ${{ github.workspace }}/.github/scripts/version-cmp.js run: | node - << 'JSEOF' const fs = require('fs'); const { execFileSync } = require('child_process'); + const { cmp } = require(process.env.VERSION_CMP_LIB); const fetchMeta = url => { try { @@ -155,14 +162,6 @@ jobs: const versionsIn = xml => [...xml.matchAll(/([^<]+)<\/version>/g)].map(m => m[1]); - const cmp = (a, b) => { - const pa = a.split('.').map(Number), pb = b.split('.').map(Number); - for (let i = 0; i < Math.max(pa.length, pb.length); i++) { - if ((pa[i] || 0) !== (pb[i] || 0)) return (pa[i] || 0) - (pb[i] || 0); - } - return 0; - }; - // Maven Central's is currently a 4.0.0 release candidate, and Dependabot // itself stays on the stable line ("Filtered out 33 pre-release versions"), so the // target is the newest stable 3.9.x rather than whatever happens to say. @@ -209,52 +208,26 @@ jobs: env: PROJECTS_FILTER: ${{ inputs.projects }} REPO_TYPE: ${{ inputs.repo_type || 'both' }} + MATRIX_LIB: ${{ github.workspace }}/.github/scripts/project-branch-matrix.js run: | node - << 'JSEOF' const fs = require('fs'); + const { buildBranchMatrix } = require(process.env.MATRIX_LIB); const projects = JSON.parse(fs.readFileSync('config/projects.json', 'utf8')); - const filterRaw = (process.env.PROJECTS_FILTER || '').trim(); - const filter = filterRaw - ? new Set(filterRaw.split(',').map(p => p.trim()).filter(Boolean)) - : new Set(); - - const repoType = (process.env.REPO_TYPE || 'both').trim(); - const typeKeys = repoType === 'both' ? ['oss', 'commercial'] : [repoType]; - // One entry per branch, not per repository - the wrapper is a file in the tree, so // every maintained branch has its own copy and its own PR. docs-build is // deliberately absent: it is not in projects.json, and its Dependabot config runs // the npm ecosystem rather than maven, so its wrapper is not part of this problem. - const entries = []; - const excluded = []; - for (const [projectKey, config] of Object.entries(projects)) { - if (projectKey === 'defaults') continue; - if (filter.size > 0 && !filter.has(projectKey)) continue; - for (const typeKey of typeKeys) { - if (!config[typeKey]) continue; - const repo = typeKey === 'commercial' - ? `spring-cloud/${projectKey}-commercial` - : `spring-cloud/${projectKey}`; - for (const branch of config[typeKey]?.branches?.scheduled || []) { - // `-internal` branches are left alone: they are the in-flight development - // line for the next train, so an unsolicited toolchain change there lands - // in the middle of active work rather than on a settled branch. They are - // filtered here rather than skipped inside the job so no runner is spent - // on them, and they are reported below so the omission stays visible. - if (branch.endsWith('-internal')) { - excluded.push(`${repo}@${branch}`); - continue; - } - entries.push({ project: projectKey, repo, type: typeKey, branch }); - } - } - } - - entries.sort((a, b) => - a.repo.localeCompare(b.repo) || a.branch.localeCompare(b.branch)); - excluded.sort(); + // `-internal` branches are excluded by default (they are the in-flight development + // line for the next train, so an unsolicited toolchain change there lands in the + // middle of active work rather than on a settled branch) and reported separately so + // the omission stays visible, rather than silently spending no runner on them. + const { entries, excluded } = buildBranchMatrix(projects, { + filter: process.env.PROJECTS_FILTER, + repoType: process.env.REPO_TYPE, + }); console.log(`Repo/branch combinations to check: ${entries.length}`); for (const e of entries) console.log(` ${e.repo}@${e.branch}`); @@ -278,8 +251,9 @@ jobs: max-parallel: 8 matrix: ${{ fromJson(needs.setup.outputs.matrix) }} steps: - # This repository, for .github/scripts/maven-wrapper-properties.js - the rules both - # this step and the regenerate step below edit properties files by. Checked out into + # This repository, for the shared .github/scripts/*.js modules - maven-wrapper-properties.js + # (the rules both this step and the regenerate step below edit properties files by), + # gh-cli.js and git-pr-helpers.js (the generic gh/git-data-API mechanics). Checked out into # a subdirectory on purpose: a checkout at the workspace root would clean the # wrapper-*.json result file the steps here write there. - name: Checkout shared scripts @@ -292,6 +266,8 @@ jobs: env: GH_TOKEN: ${{ inputs.token || secrets.GH_ACTIONS_REPO_TOKEN }} WRAPPER_LIB: ${{ github.workspace }}/ci-actions/.github/scripts/maven-wrapper-properties.js + GH_CLI_LIB: ${{ github.workspace }}/ci-actions/.github/scripts/gh-cli.js + GIT_PR_LIB: ${{ github.workspace }}/ci-actions/.github/scripts/git-pr-helpers.js REPO: ${{ matrix.repo }} BRANCH: ${{ matrix.branch }} PROJECT: ${{ matrix.project }} @@ -308,7 +284,11 @@ jobs: run: | node - << 'JSEOF' const fs = require('fs'); - const { execFileSync } = require('child_process'); + + // The generic gh-CLI and git-data-API/PR mechanics - shared with + // update-antora-ui-bundle.yml, which opens bot PRs the same way. + const { ghRetry } = require(process.env.GH_CLI_LIB); + const PR = require(process.env.GIT_PR_LIB); // Same rules the regenerate step applies against a real checkout - see the header of that // file for why they live outside this workflow. @@ -326,33 +306,6 @@ jobs: // Audit mode: report what Dependabot would fail to parse, and write nothing at all. const CHECK = process.env.CHECK_ONLY === 'true'; - const gh = args => { - try { - return { ok: true, out: execFileSync('gh', args, - { encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'], maxBuffer: 1 << 26 }) }; - } catch (err) { - return { ok: false, out: err.stdout || '', - err: (err.stderr || err.message || '').split('\n')[0].trim() }; - } - }; - const ghRetry = (args, attempts = 3) => { - let last; - for (let i = 1; i <= attempts; i++) { - last = gh(args); - if (last.ok) return last; - if (i < attempts) execFileSync('sleep', [String(i * 3)]); - } - return last; - }; - - // Write requests go through --input with a JSON file rather than repeated -f - // flags: -f cannot express the nested author/committer objects the contents API - // needs, and it would mangle the newlines in a commit message or PR body. - const ghJson = (path, method, payload) => { - fs.writeFileSync('payload.json', JSON.stringify(payload)); - return ghRetry(['api', path, '--method', method, '--input', 'payload.json']); - }; - const safe = `${REPO}@${BRANCH}`.replace(/[/@]/g, '-'); const finish = (status, detail, extra = {}) => { const result = { project: process.env.PROJECT, repo: REPO, type: process.env.TYPE, @@ -400,17 +353,10 @@ jobs: { truncated }); } - const readAt = (path, ref) => { - const r = gh(['api', `repos/${REPO}/contents/${path}?ref=${ref}`]); - if (!r.ok) return null; - const meta = JSON.parse(r.out); - return { sha: meta.sha, text: Buffer.from(meta.content, 'base64').toString('utf8') }; - }; - const files = []; for (const dir of dirs) { const path = W.propsPath(dir); - const got = readAt(path, enc); + const got = PR.readFileAt(REPO, path, enc); if (!got) finish('error', `could not read ${path}`); files.push({ dir, path, sha: got.sha, text: got.text }); } @@ -424,7 +370,7 @@ jobs: for (const f of files) { if (W.dependabotWrapperVersion(f.text)) continue; const present = W.scriptPaths(f.dir).filter(p => pathSet.has(p)); - const scripts = present.map(p => readAt(p, enc)).filter(Boolean).map(s => s.text); + const scripts = present.map(p => PR.readFileAt(REPO, p, enc)).filter(Boolean).map(s => s.text); if (W.dependabotWrapperVersion(f.text, scripts)) continue; broken.push({ path: f.path, reason: present.length ? `no wrapperVersion or wrapperUrl, and no version banner in ${present.join(', ')}` @@ -500,53 +446,23 @@ jobs: // explicitly here rather than left to the token's default identity. const message = `Update Maven wrapper to ${MAVEN}\n\n` + `Signed-off-by: ${authorName} <${authorEmail}>`; + const author = { name: authorName, email: authorEmail }; - // One commit for N files. The contents API writes a single file per call and therefore - // makes a separate commit for each, so a branch touching nine module wrappers would arrive - // as nine commits with the same message. The git data API builds one tree and one commit - // instead, which is also the shape the regenerate path produces. - const commitFiles = (ref, list) => { - const refSha = ghRetry(['api', `repos/${REPO}/git/ref/heads/${ref}`, '--jq', '.object.sha']); - if (!refSha.ok) return { ok: false, err: `could not read ${ref}: ${refSha.err}` }; - const parent = refSha.out.trim(); - - const baseTree = ghRetry(['api', `repos/${REPO}/git/commits/${parent}`, '--jq', '.tree.sha']); - if (!baseTree.ok) return { ok: false, err: `could not read the tree of ${ref}: ${baseTree.err}` }; - - // mode 100644: maven-wrapper.properties is never executable. Everything not listed is - // inherited from base_tree, so file modes elsewhere in the repository are untouched. - const made = ghJson(`repos/${REPO}/git/trees`, 'POST', { - base_tree: baseTree.out.trim(), - tree: list.map(f => ({ path: f.path, mode: '100644', type: 'blob', content: f.updated })), - }); - if (!made.ok) return { ok: false, err: `could not build a tree: ${made.err}` }; - - const commit = ghJson(`repos/${REPO}/git/commits`, 'POST', { - message, - tree: JSON.parse(made.out).sha, - parents: [parent], - author: { name: authorName, email: authorEmail }, - committer: { name: authorName, email: authorEmail }, - }); - if (!commit.ok) return { ok: false, err: `could not create a commit: ${commit.err}` }; - - const moved = ghJson(`repos/${REPO}/git/refs/heads/${ref}`, 'PATCH', - { sha: JSON.parse(commit.out).sha }); - if (!moved.ok) return { ok: false, err: `could not move ${ref}: ${moved.err}` }; - return { ok: true }; - }; + // One commit for N files, via git-pr-helpers.js's tree/commit/ref-move sequence - + // the contents API writes a single file per call and therefore makes a separate + // commit for each, so a branch touching nine module wrappers would arrive as nine + // commits with the same message. + const commitFiles = (ref, list) => PR.commitFilesToRef(REPO, ref, + list.map(f => ({ path: f.path, content: f.updated })), message, author); // Matched by prefix and base rather than by exact head, so a PR opened for an // earlier target version is found too. Otherwise each new Maven release would // stack another PR on top of the last one, all editing the same file. - const openPrs = ghRetry(['api', - `repos/${REPO}/pulls?state=open&base=${enc}&per_page=100`]); - if (!openPrs.ok) finish('error', `could not list pull requests: ${openPrs.err}`); - const mine = JSON.parse(openPrs.out || '[]') - .filter(pr => (pr.head?.ref || '').startsWith(`${PREFIX}${slug}-`)); - - if (mine.length) { - const pr = mine[0]; + const findPr = PR.findOpenPrByHeadPrefix(REPO, BRANCH, `${PREFIX}${slug}-`); + if (!findPr.ok) finish('error', `could not list pull requests: ${findPr.err}`); + + if (findPr.pr) { + const pr = findPr.pr; // Read the PR's own head branch: it may already be at the target (nothing to // do), or behind it (a newer Maven has shipped since it was opened). The file list // comes from the base branch, so a wrapper added to the base after the PR was opened is @@ -554,7 +470,7 @@ jobs: const headRef = encodeURIComponent(pr.head.ref); const onHead = []; for (const f of planned) { - const got = readAt(f.path, headRef); + const got = PR.readFileAt(REPO, f.path, headRef); if (got) onHead.push({ ...f, sha: got.sha, text: got.text }); } const headPlan = plan(onHead); @@ -599,18 +515,11 @@ jobs: } // ── Create the branch ─────────────────────────────────────────────────────────── - const baseRef = ghRetry(['api', `repos/${REPO}/git/ref/heads/${BRANCH}`, '--jq', '.object.sha']); - if (!baseRef.ok) finish('error', `could not read ${BRANCH}: ${baseRef.err}`); - const baseSha = baseRef.out.trim(); - - const madeRef = ghJson(`repos/${REPO}/git/refs`, 'POST', - { ref: `refs/heads/${head}`, sha: baseSha }); - if (!madeRef.ok && !/already exists/i.test(madeRef.err)) { - finish('error', `could not create ${head}: ${madeRef.err}`); - } + const madeRef = PR.createBranch(REPO, head, BRANCH); + if (!madeRef.ok) finish('error', madeRef.err); // An existing branch with no open PR means a previous PR was closed unmerged. // Leaving it alone is deliberate - reopening it would re-litigate that decision. - if (!madeRef.ok) { + if (madeRef.alreadyExists) { finish('branch-exists', `${head} exists but no PR is open - left alone`, extras); } @@ -649,15 +558,14 @@ jobs: 'repository\'s update job - not merely the wrapper update.'); } - const pr = ghJson(`repos/${REPO}/pulls`, 'POST', { + const opened = PR.openPr(REPO, { title: `Update Maven wrapper to ${MAVEN}`, head, base: BRANCH, body: prBody.join('\n'), }); - if (!pr.ok) finish('error', `could not open PR: ${pr.err}`); + if (!opened.ok) finish('error', opened.err); - const created = JSON.parse(pr.out); - finish('pr-opened', `#${created.number}: ${versionPhrase(changed)} (${describe(changed)})`, - { ...extras, prUrl: created.html_url, prNumber: created.number }); + finish('pr-opened', `#${opened.pr.number}: ${versionPhrase(changed)} (${describe(changed)})`, + { ...extras, prUrl: opened.pr.html_url, prNumber: opened.pr.number }); JSEOF # ── Regenerate mode ───────────────────────────────────────────────────────────── @@ -915,10 +823,11 @@ jobs: SAFE_NAME: ${{ steps.update.outputs.safe-name }} MERGE_METHOD: ${{ inputs.merge_method || 'squash' }} DRY_RUN: ${{ github.event_name == 'schedule' && 'false' || (inputs.dry_run != false) }} + MERGE_LIB: ${{ github.workspace }}/ci-actions/.github/scripts/merge-if-green.js run: | node - << 'JSEOF' const fs = require('fs'); - const { execFileSync } = require('child_process'); + const { mergeIfGreen } = require(process.env.MERGE_LIB); const REPO = process.env.REPO; const PR = process.env.PR_NUMBER; @@ -926,70 +835,15 @@ jobs: const METHOD = process.env.MERGE_METHOD || 'squash'; const FILE = `wrapper-${process.env.SAFE_NAME}.json`; - const gh = args => { - try { - return { ok: true, out: execFileSync('gh', args, - { encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'], maxBuffer: 1 << 26 }) }; - } catch (err) { - return { ok: false, out: err.stdout || '', - err: (err.stderr || err.message || '').split('\n')[0].trim() }; - } - }; + const { status, detail } = mergeIfGreen({ repo: REPO, pr: PR, method: METHOD, dryRun: DRY }); // Merge outcome is folded back into the result the previous step wrote, so the // summary reports one row per repo/branch rather than two disjoint views. - const record = (mergeStatus, mergeDetail) => { - let result = {}; - try { result = JSON.parse(fs.readFileSync(FILE, 'utf8')); } catch (_) { /* first write */ } - fs.writeFileSync(FILE, JSON.stringify({ ...result, mergeStatus, mergeDetail }, null, 2)); - console.log(`${REPO} #${PR}: ${mergeStatus}${mergeDetail ? ` - ${mergeDetail}` : ''}`); - process.exit(0); - }; - - const view = gh(['pr', 'view', PR, '--repo', REPO, '--json', - 'mergeable,mergeStateStatus,statusCheckRollup,url']); - if (!view.ok) record('error', `could not read PR: ${view.err}`); - const pr = JSON.parse(view.out); - - // Checks are read from statusCheckRollup rather than inferred from - // mergeStateStatus, which conflates failing checks with "needs review" and a - // locked branch - the same distinction dependabot-report.yml makes. - const FAILING = new Set(['FAILURE', 'TIMED_OUT', 'ERROR', 'STARTUP_FAILURE', 'CANCELLED']); - const PASSING = new Set(['SUCCESS', 'NEUTRAL', 'SKIPPED']); - const checks = (pr.statusCheckRollup || []).map(c => { - if (c.__typename === 'StatusContext' || c.state) { - return { name: c.context || 'status', state: (c.state || '').toUpperCase() }; - } - const status = (c.status || '').toUpperCase(); - if (status && status !== 'COMPLETED') return { name: c.name, state: 'PENDING' }; - return { name: c.name, state: (c.conclusion || 'PENDING').toUpperCase() }; - }); - - const failing = checks.filter(c => FAILING.has(c.state)); - const pending = checks.filter(c => !FAILING.has(c.state) && !PASSING.has(c.state)); - - if (!checks.length) record('not-merged', 'no checks have reported yet'); - if (failing.length) { - record('not-merged', `failing checks: ${failing.map(c => c.name).join(', ')}`); - } - if (pending.length) { - record('not-merged', `checks still running: ${pending.map(c => c.name).join(', ')}`); - } - if (pr.mergeable === 'CONFLICTING') record('not-merged', 'conflicts with the base branch'); - if (pr.mergeable !== 'MERGEABLE') record('not-merged', `mergeable is ${pr.mergeable}`); - - // Green but not CLEAN means branch protection is holding it - a required review, - // or a Release Freeze on the base branch. The merge API would refuse anyway, so - // this is reported rather than attempted. - if (pr.mergeStateStatus !== 'CLEAN') { - record('not-merged', `all checks pass but GitHub reports ${pr.mergeStateStatus}`); - } - - if (DRY) record('would-merge', `all ${checks.length} check(s) pass`); - - const merged = gh(['pr', 'merge', PR, '--repo', REPO, `--${METHOD}`, '--delete-branch']); - if (!merged.ok) record('error', `merge failed: ${merged.err}`); - record('merged', `${METHOD}, all ${checks.length} check(s) passed`); + let result = {}; + try { result = JSON.parse(fs.readFileSync(FILE, 'utf8')); } catch (_) { /* first write */ } + fs.writeFileSync(FILE, JSON.stringify( + { ...result, mergeStatus: status, mergeDetail: detail }, null, 2)); + console.log(`${REPO} #${PR}: ${status}${detail ? ` - ${detail}` : ''}`); JSEOF # Guarded on safe-name: if the script died before writing its result, the artifact diff --git a/docs/release-automation.md b/docs/release-automation.md index 4edfec0..8869eae 100644 --- a/docs/release-automation.md +++ b/docs/release-automation.md @@ -874,6 +874,7 @@ See [DESIGN-dependabot-automation.md](../DESIGN-dependabot-automation.md) for th |---|---| | [`rollout-actions-ref.yml`](../.github/workflows/README-rollout-actions-ref.md) | Repoints every consumer's reference to this repo at a released SHA, via [`resolve-actions-ref`](../.github/actions/resolve-actions-ref/) (latest release → SHA + tag) and [`sync-actions-ref`](../.github/actions/sync-actions-ref/) (idempotent per-branch rewrite). Weekly scheduled run is a **forced dry run** that reports drift only | | [`update-maven-wrapper.yml`](../.github/workflows/README-update-maven-wrapper.md) | Weekly wrapper check across every repo/branch, opening a PR where behind. Finds *every* `maven-wrapper.properties` including submodules, because Dependabot aborts a repo's whole update job on one unreadable file | +| [`update-antora-ui-bundle.yml`](../.github/workflows/README-update-antora-ui-bundle.md) | Weekly check of every repo/branch's `docs/antora-playbook.yml` against the latest `spring-io/antora-ui-spring` release, opening a PR where behind and posting to Chat when anything changed | | [`ci-status-report.yml`](../.github/workflows/README-ci-status-report.md) | Two-phase: ~4am reruns failing jobs silently, ~6am rescans, blames the breaking commit, reports to Chat. Always exits 0 | | [`retire-branch.yml`](../.github/workflows/README-retire-branch.md) | Removes a branch from `projects.json`, drops its Dependabot entries and playbook entry, locks it permanently via the `Locked Branches` ruleset | | [`check-token-permissions.yml`](../.github/workflows/README-check-token-permissions.md) | Probes a token for every permission the automation needs. Run after rotating `GH_ACTIONS_REPO_TOKEN` | @@ -897,8 +898,10 @@ The topology source of truth. Per project: (`spring-cloud-cloudfoundry`, `spring-cloud-sleuth`) are commercial-only legacy. Read by `determine-matrix`, every `rollout-*` workflow, `update-maven-wrapper`, -`ci-status-report`, and both Dependabot workflows. Written by `add-branches-projects-json`, -`update-projects-json`, and `retire-branch-projects-json`. +`update-antora-ui-bundle`, `ci-status-report`, `lock-unlock-branches`, and both Dependabot +workflows — via the shared `project-branch-matrix.js` walk in most of those, rather than each +carrying its own copy. Written by `add-branches-projects-json`, `update-projects-json`, and +`retire-branch-projects-json`. ### Release-train action overrides @@ -922,7 +925,7 @@ on every push and pull request to this repository: - **`test-*.yml`** — one workflow per action with meaningful logic (`test-extract-bom-versions`, `test-update-project-versions`, `test-verify-no-snapshot-versions`, `test-spring-release-train-project-ready`, - `test-maven-wrapper-properties`), exercising it against fixtures. + `test-shared-scripts`), exercising it against fixtures. - **[`verify-dist.yml`](../.github/workflows/verify-dist.yml)** — discovers every action with a `src/` directory, rebuilds it, and fails if the committed `dist/` does not match the source. This gates every release, so a stale bundle can never be tagged.