Skip to content

Harden CI workflow: main-only push, concurrency, timeouts, SHA pins, install retry, Python 3.14 - #107

Merged
CompleteDotTech merged 2 commits into
mainfrom
ci/50-harden-workflow
Nov 13, 2026
Merged

CompleteDotTech merged 2 commits into
mainfrom
ci/50-harden-workflow

Conversation

@CompleteDotTech

@CompleteDotTech CompleteDotTech commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Issue #50 recorded CI hygiene gaps that the owner has now approved fixing: every branch push and its pull request ran duplicate checks, superseded runs were never cancelled, jobs had no timeout (360 minute default), actions/checkout and actions/setup-python used mutable tags, and one hosted run on main failed once at dependency install with no retry.

Change

.github/workflows/ci.yml and the new scripts/ci_pip_install.sh change behavior. Alongside them: a CHANGELOG entry, the current CI matrix sentence in docs/HIERARCHY_RELEASE_VERIFICATION.md (3.11–3.13 → 3.11–3.14, historical evidence lines left alone), and MANIFEST.sha256 entries for ci.yml (refreshed) and scripts/ci_pip_install.sh (added).

  1. Trigger: push is limited to branches: [main]. pull_request is unchanged.
  2. Concurrency: group ${{ github.workflow }}-${{ github.event_name == 'pull_request' && format('pr-{0}', github.event.pull_request.number) || github.run_id }}, cancel-in-progress: ${{ github.event_name == 'pull_request' }}.
    • Pull request runs share one group per PR, so a newer push cancels the superseded run.
    • Every other run (pushes to main) gets a group of its own. GitHub keeps at most one running and one pending run per group and cancels the pending one when a newer run joins, whatever cancel-in-progress says. A shared main group could therefore drop the run for a commit merged in quick succession. With a unique group per run, main runs are never queued behind or cancelled by another run.
    • Why run_id and not sha: run_id is unique per workflow run and stays the same across re-runs (only run_attempt changes), and attempts of one run never overlap. So no two distinct runs ever share a group. A sha key would put two runs of the same commit in one group (for example a re-run started while another run of that commit is still going), which brings back the pending-cancel case.
    • (Round 1 used github.event.pull_request.number || github.ref, which had the shared-main-group problem above.)
  3. Timeouts: timeout-minutes: 20 on core and optional-contracts. The last three successful runs before this PR (37524953457, 37128382459, 37128039600) had jobs that each finished in 0.6–1.1 minutes. 20 minutes allows for worst-case install retries (2 × 15 s sleeps per install step) and slow runners.
  4. Action pins (latest release in the major version already used; each major tag currently points at the same commit, so the code that runs does not change):
    • actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
    • actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
  5. Install retry: all six pip install steps (core: .[dev], example requirements, and the installed-wheel venv install; optional: .[all,dev], example requirements, pinned DSPy/GEPA) call bash scripts/ci_pip_install.sh <python> <pip args...>. It runs <python> -m pip install up to 3 times with a 15 s sleep between attempts. If all 3 fail, it logs ::error:: and exits 1, failing the step. The retry policy now lives in one tracked file (it sits next to the existing scripts/setup.sh), and no new action dependency was added. pip's exit code does not tell network failures apart from resolver errors, so every failure is still retried. A deterministic error costs about 30 s before the step fails. Classifying pip output was left out on purpose, because it would be fragile.
  6. Python 3.14 is added to the core matrix.

Pin verification

$ gh api repos/actions/checkout/git/ref/tags/v4.4.0
{"ref":"refs/tags/v4.4.0", ... "object":{"sha":"11d5960a326750d5838078e36cf38b85af677262","type":"commit", ...}}
$ gh api repos/actions/checkout/git/ref/tags/v4
{"ref":"refs/tags/v4", ... "object":{"sha":"11d5960a326750d5838078e36cf38b85af677262","type":"commit", ...}}
$ gh api repos/actions/setup-python/git/ref/tags/v5.6.0
{"ref":"refs/tags/v5.6.0", ... "object":{"sha":"a26af69be951a213d495a4c3e4e4022e16d87065","type":"commit", ...}}
$ gh api repos/actions/setup-python/git/ref/tags/v5
{"ref":"refs/tags/v5", ... "object":{"sha":"a26af69be951a213d495a4c3e4e4022e16d87065","type":"commit", ...}}

All four refs are lightweight tags (type: commit), so no tag object needed dereferencing. gh release list lists v4.4.0 (checkout) and v5.6.0 (setup-python) as the newest releases in those majors. Moving to newer majors (checkout v7, setup-python v7) was left out of scope.

Final check names

Job names are unchanged. Matrix expansion adds one core check:

  • core (3.11)
  • core (3.12)
  • core (3.13)
  • core (3.14) (new)
  • optional-contracts (3.3.1)
  • optional-contracts (3.4.0)

Validation (head e1d6d97)

  • Rebased onto current origin/main (3859256, after Validate gateway destinations before credentialed dispatch #98). The CHANGELOG conflict was resolved by keeping both entries.
  • YAML parses with yaml.safe_load. Parsed trigger: {'push': {'branches': ['main']}, 'pull_request': None}. Concurrency group as above. Timeouts: core 20, optional-contracts 20. Matrices: core 3.11/3.12/3.13/3.14, optional dspy 3.3.1/3.4.0.
  • I simulated scripts/ci_pip_install.sh locally under bash --noprofile --norc -eo pipefail with a stub interpreter in place of pip, and the sleep set to 0:
    • Success on attempt 1, 2 or 3 gives exit 0, and the script stops after 1, 2 or 3 calls.
    • Three failures give ::error::pip install failed after 3 attempts and exit 1, after exactly 3 calls.
    • Fewer than 2 arguments prints usage and exits 2.
  • shellcheck and actionlint are not installed locally, so neither was run.
  • Local Python 3.14.8, optional SDKs (dspy/gepa/typesafe-sdk) absent: pytest -q 737 passed, 79 skipped (the count rose from 606 because Validate gateway destinations before credentialed dispatch #98 added tests on main). ruff check src tests examples: all checks passed.
  • sha256sum -c MANIFEST.sha256 reports OK for .github/workflows/ci.yml and scripts/ci_pip_install.sh. The rest of MANIFEST.sha256 was already out of date on main (including scripts/setup.sh and scripts/setup.ps1). This PR leaves those entries alone.
  • Hosted checks on this head: run 43355642084 on head e1d6d97 (it is the PR head commit's status rollup). All 6 checks succeeded:
    • core (3.11) pass (54s)
    • core (3.12) pass (49s)
    • core (3.13) pass (45s)
    • core (3.14) pass (31s)
    • optional-contracts (3.3.1) pass (50s)
    • optional-contracts (3.4.0) pass (59s)
    • I did not read the per-job pytest counts or check for retry warnings in this round, because the GitHub REST rate limit was exhausted and job logs need REST.

Not executed

Refs #50

🤖 Generated with Claude Code

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-13T03:19:46.011057Z 7b7f13b New commits
🔒 Security Review ✅ Completed 2026-11-13T02:49:46.357556Z 2d8ed8b PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@CompleteDotTech

Copy link
Copy Markdown
Contributor Author

Fresh-context review (round 1) of head 2d8ed8b

What I checked myself

  • Local run on this head (Python 3.14.8; dspy, gepa and typesafe-sdk not installed): pytest -q gave 606 passed, 79 skipped, the same as the baseline on main 6532cc8. ruff check src tests examples passed.
  • MANIFEST.sha256: sha256sum .github/workflows/ci.yml = cea8dce8…2968, which matches the updated entry. No other entry was changed.
  • Action pins: I checked these with gh api repos/actions/<repo>/tags and /commits/<sha>:
    • checkout v4.4.0 and v4 both point to 11d5960a326750d5838078e36cf38b85af677262.
    • setup-python v5.6.0 and v5 both point to a26af69be951a213d495a4c3e4e4022e16d87065.
    • Both commits exist upstream. The releases API lists v4.4.0 and v5.6.0 as the newest releases in their majors.
  • Retry loop: I simulated it under GitHub's default bash --noprofile --norc -eo pipefail, with pip replaced by a stub:
    • Success on attempt 1, 2 or 3 gives exit 0, and the loop stops right after the successful attempt.
    • Three failures give ::error:: and exit 1.
    • The failing command sits inside if, so -e does not end the step early. The final failure is not hidden.
    • The installed-wheel step keeps the rest of its script after the loop, so later failures still fail the step.
  • Hosted checks on the exact head: all 6 check runs on 2d8ed8b completed with success: core 3.11, 3.12, 3.13 and 3.14, and optional-contracts 3.3.1 and 3.4.0. There is only one workflow run for this SHA (43354539279, pull_request), so the branch push started no duplicate run.
  • YAML: parses with yaml.safe_load. The trigger is push: {branches: [main]} plus pull_request. Both jobs have timeout-minutes: 20.
  • Invariants: no secrets, no eval/exec, and no change to the import boundary, splits, budgets or identity checks. Only CI config and docs changed.
  • Repo hygiene: CI that never ran on PRs #1-#5, no branch protection, stale branches, tracked wheel #50 CI item: "Restrict push to main" is met, and the run list above shows it working. "3.14 is not in CI" is fixed. The other changes (concurrency, timeouts, pins, retry) are hardening this PR adds on top.
  • Docs: the docs/BUILD_REPORT.md lines that say "3.11-3.13" are historical "not executed" records, so leaving them unchanged is correct.

Findings

BLOCKING: .github/workflows/ci.yml:8-11. The concurrency group can cancel pending runs on main, which the comment says never happens.

  • The comment at line 8 says "runs on main are never cancelled". The PR body says the same.
  • For push events the group is tests-refs/heads/main. cancel-in-progress: false only protects the run that is in progress.
  • GitHub's documented concurrency behaviour: while one run in a group is in progress, a new run waits as pending, and any run already pending in that group is cancelled when a newer one is queued.

Failure scenario:

  1. Commits A, B and C are merged to main within the time one run takes (about 1 minute now, up to 20 with retries).
  2. The run for A is in progress and B's run is pending.
  3. When C is pushed, B's run is cancelled. Commit B never gets a hosted run on main.
  4. Runs on main are also forced to run one at a time.

This repo cites per-commit main runs as evidence ("Fresh software evidence on main <sha>" in docs/HIERARCHY_RELEASE_VERIFICATION.md:31). So a quietly skipped main commit is a real gap, and the comment states a guarantee the config does not give.

Fix: give non-PR runs a unique group so they are never grouped or cancelled, for example:

group: ${{ github.workflow }}-${{ github.event_name == 'pull_request' && format('pr-{0}', github.event.pull_request.number) || github.sha }}

or github.run_id in place of github.sha. Then keep the comment, which would be accurate.

NIT: .github/workflows/ci.yml (six copies of the retry block).

  • The same 6-line loop is copied into every install step. A later change to one copy (attempt count, message) could drift from the others.
  • The loop also retries deterministic resolver errors, which adds about 30 s before an error that cannot pass anyway.
  • This is acceptable as is. It is only worth noting.

No other findings.

@CompleteDotTech

Copy link
Copy Markdown
Contributor Author

Round 1 findings addressed on head e1d6d97d27347974bbf334e0157248da757c0e7f, rebased onto origin/main 3859256.

BLOCKING (shared concurrency group on main): fixed.

  • The group is now ${{ github.workflow }}-${{ github.event_name == 'pull_request' && format('pr-{0}', github.event.pull_request.number) || github.run_id }}. cancel-in-progress is unchanged, so it is true only for PRs.
  • PR runs keep one group per PR, so superseded runs are still cancelled.
  • Every non-PR run gets its own group, so a pending main run can no longer be dropped when a newer one queues, and main runs are no longer forced to run one at a time.
  • Why run_id over sha: run_id is unique per run and stays the same across re-runs, and attempts of one run never overlap. A sha key would put two runs of the same commit in one group, so a re-run started while another run of that commit is in progress could bring back the pending-cancel case.
  • The workflow comment now says exactly this.

NIT (six copied retry loops): fixed (deduplicated). Retrying non-transient errors is kept and documented.

  • All six install steps now call bash scripts/ci_pip_install.sh <python> <pip args...>. This follows the existing scripts/setup.sh convention and adds no new action dependency.
  • pip's exit code does not tell network failures apart from resolver errors, so every failure is still retried, at a cost of about 30 s for a deterministic error. I chose not to parse pip output to classify errors, because that would be fragile. The script header and the PR body note this.
  • I simulated the script locally under bash -eo pipefail with a stub interpreter:
    • Success on attempt 1, 2 or 3 gives exit 0 after 1, 2 or 3 calls.
    • Three failures give ::error:: and exit 1.
    • Calling it with no pip arguments exits 2.

Local validation: pytest -q gave 737 passed and 79 skipped. ruff check src tests examples passed. The YAML parses. The MANIFEST.sha256 entries for ci.yml and the new script are OK. Hosted results will be added to the PR body once the checks finish.

@CompleteDotTech

Copy link
Copy Markdown
Contributor Author

Fresh-context review (round 2) of head e1d6d97

Scope: #50, the CI-workflow hardening item only. I read the diff, the full ci.yml, the new script and the touched docs, and did not rely on the PR description.

What I verified myself

  • Local run on the PR head (Python 3.14.8, dspy/gepa/typesafe-sdk absent):
    • pytest -q: 737 passed, 79 skipped, the same as the origin/main 3859256 baseline.
    • ruff check src tests examples: clean.
  • MANIFEST.sha256: sha256sum -c reports OK for .github/workflows/ci.yml and scripts/ci_pip_install.sh. No other entries changed. schemas/ and fixtures are untouched.
  • Action pins, checked with git ls-remote against upstream rather than the REST API:
    • actions/checkout: refs/tags/v4.4.0 and refs/tags/v4 both point to 11d5960a326750d5838078e36cf38b85af677262. There is no ^{} peeled entry, so these are lightweight tags pointing at the commit. v4.4.0 is the newest v4.x.y tag.
    • actions/setup-python: refs/tags/v5.6.0 and refs/tags/v5 both point to a26af69be951a213d495a4c3e4e4022e16d87065, also lightweight. v5.6.0 is the newest v5.x.y tag.
  • Concurrency expression:
    • For pull_request events, format('pr-{0}', number) is a non-empty string, so the &&/|| expression yields tests-pr-<N>. cancel-in-progress evaluates to true, so a newer push cancels the superseded run.
    • For any other event (now only push to main), the group is tests-<run_id>. That group is unique per run and stays the same across re-run attempts, and attempts never overlap. A main run therefore never shares a group, never sits as a pending run that a newer run could replace, and is never cancelled (cancel-in-progress is false).
  • Retry script (scripts/ci_pip_install.sh). I simulated it with a stub interpreter, the sleep set to 0, under bash --noprofile --norc -eo pipefail:
    • Success on the first attempt gives exit 0 after 1 call.
    • Success on the third attempt gives exit 0 after 3 calls, with 2 ::warning:: lines.
    • Three failures give ::error:: and exit 1 after exactly 3 calls.
    • Fewer than 2 arguments gives the usage message and exit 2.
    • Arguments containing spaces or brackets ('a b', '.[dev]', 'dspy[litellm]==3.4.0') reach the interpreter as single arguments, because "$@" is used.
    • The failing pip runs as an if condition, so set -e does not abort the loop.
    • In the installed-wheel step, the call comes before cd "$temp_dir", so the relative scripts/ path resolves. The wheel glob is expanded by the step shell.
  • Hosted checks on the exact head:
    • Over GraphQL, the commit e1d6d97… has exactly one github-actions check suite: run 43355642084, event pull_request, 6/6 checks SUCCESS. There is no duplicate push run for the branch, which is consistent with the new push filter.
    • Over REST, the run reports head_sha e1d6d97, conclusion success.
    • From the job logs:
      • setup-python installed CPython 3.11.18, 3.12.16, 3.13.18 and 3.14.10.
      • Each core job reports 737 passed, 79 skipped. The optional jobs report 765 passed, 51 skipped and 783 passed, 33 skipped.
      • No pip install attempt … failed warnings appear, so the retry path was not exercised on hosted CI.
  • AGENTS.md invariants: no source, runtime, split, budget or identity code is touched. No secrets, eval/exec or pickle are added. The docs change (docs/HIERARCHY_RELEASE_VERIFICATION.md:27) describes the configured matrix and keeps the existing "a green configuration file alone proves nothing" caveat. The historical 3.11–3.13 statements in docs/BUILD_REPORT.md are dated evidence and are correctly left alone.

Findings

  • NIT (.github/workflows/ci.yml:39): - run: python -m build still installs dependencies from the network without a retry. build creates an isolated environment and pip-installs the [build-system].requires packages (setuptools>=77, wheel) from PyPI. A transient PyPI or network failure at that step fails core just as the original Repo hygiene: CI that never ran on PRs #1-#5, no branch protection, stale branches, tracked wheel #50 install failure did, so "bounded retry around dependency installs" is not complete for this one step. Possible fixes:

    • Wrap it in a small retry.
    • Or pre-install setuptools>=77 wheel through ci_pip_install.sh and run python -m build --no-isolation.

    This step is not one of the six pip install steps the PR lists, so it is not a regression, and the main install steps are covered.

No BLOCKING findings.

REVIEW CLEAN

CompleteDotTech and others added 2 commits November 12, 2026 22:17
Limit push runs to main, cancel superseded pull request runs, set
20 minute job timeouts, pin checkout and setup-python to verified
commit SHAs, retry pip installs up to three times, and add Python
3.14 to the core matrix.

Refs #50

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The shared push group let GitHub cancel a pending main run when a newer
one queued, despite cancel-in-progress being false. Non-PR runs now use
github.run_id as their group key, so main runs are never queued or
cancelled; PR runs keep one group per PR with cancellation.

The six copied pip retry loops are replaced by scripts/ci_pip_install.sh.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@CompleteDotTech

Copy link
Copy Markdown
Contributor Author

Coordinator: approved for merge. Review round 2 posted REVIEW CLEAN on e1d6d97. Round 1 found one BLOCKING item, the shared main concurrency group, and it was fixed in e1d6d97. As merger I then rebased onto origin/main 43903f0, after #106 merged. The only conflict was CHANGELOG ordering, resolved by keeping both entries; MANIFEST merged cleanly, and every touched entry checks OK. That gives head 7b7f13b: 741 passed, 79 skipped and ruff clean locally, with all 6 hosted checks passing on this exact head (run 43358063986). Because the rebase only reordered changelog lines, no extra review round was run.

Round-2 NIT left as noted, not fixed: the python -m build step (ci.yml:39) still downloads setuptools/wheel with no retry. It was never one of the install steps this PR covers, so it isn't a regression.

@CompleteDotTech
CompleteDotTech merged commit c8e0e38 into main Nov 13, 2026
6 checks passed
@CompleteDotTech
CompleteDotTech deleted the ci/50-harden-workflow branch November 13, 2026 03:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant