CI: publish releases to PyPI by rok · Pull Request #107 · benchx-dev/benchx

rok · 2026-11-06T16:00:36Z

Adds automated publishing to PyPI, triggered by v* tags. Part of #105, which has the plan.

PyPI is the only registry this repository publishes to, using trusted publishing, so no token is stored here. Conda packages will come from a conda-forge feedstock, which follows the releases on PyPI. Submitting that feedstock is separate work and needs the first PyPI release, so it isn't part of this PR.

What changes

.github/workflows/release.yml (new): builds and tests the sdist and wheel, uploads to PyPI, and creates the GitHub release. Only a tag publishes: a pull request that changes the packaging or the version runs only the build job.
docs/releasing.md (new): the release checklist, including the conda-forge step, starting the next .dev0 version, and what to do if a run fails.
pyproject.toml, uv.lock: the version is now read from src/benchx/__init__.py, so it's kept in one place.
src/benchx/runner.py: renames the runner from benchx-prototype to benchx, since that name is written into every result document.
docs/benchx.md: adds the pip install instruction and makes the UC-04 link absolute, because this page is the PyPI description.

Testing

On GitHub: the build job passed on earlier commits of this PR, and it runs again on the current head.
Tests, run locally at 7e1d6b8: the wheel on Python 3.13 and the editable install each gave 545 passed and 6 skipped. The later commits change only the workflow and docs.
Current head, run locally: twine check --strict passes, the strict docs build reports no issues, and actionlint and zizmor (offline) report nothing, with 6 zizmor findings suppressed by its defaults that I didn't inspect.

Not tested yet: the approval, PyPI trusted publishing and creating the GitHub release can't run from a pull request. They first run with the v0.1.0rc1 tag, so a problem there would need a follow-up fix and a new release candidate.

Before the first release

A maintainer needs to set up:

PyPI: a pending publisher on the benchx org: owner benchx-dev, repository benchx, workflow release.yml, environment pypi.
GitHub: the pypi environment already exists, with administrator bypass off and self-review prevention on, so a release takes two people. Check that its v* rule is a tag rule: it was first added as a branch rule, which should make GitHub reject the publish job on a tag. Automate releases to PyPI #105 proposes the PyPI org members as required reviewers, while the environment lists six people, so that needs settling.

The docs site deploys on merge, so the new install command won't work until the first release.

The version was in both pyproject.toml and src/benchx/__init__.py. A release needs one value, since the tag, the packages and the version recorded in results must all agree.

The name is written into every result document, so change it before the first release.

This page is the package's description on PyPI, where a relative link does not resolve.

A v* tag builds and tests the sdist, the wheel and a noarch conda package, then uploads to PyPI through trusted publishing and to the BenchX channel on anaconda.org, and creates the GitHub release. A pull request that changes the packaging runs the build jobs only.

PyPI's trusted publishing needs no stored token, while uploading to anaconda.org would need a long-lived one. Keep building and testing the conda package, which publishes nothing: it tests the oldest supported Python, and the recipe can seed a conda-forge feedstock.

Conda packages will come from a feedstock on conda-forge, which has its own recipe and follows the releases on PyPI. Remove the recipe and the conda job here: they would duplicate it, and a conda failure should not hold up a PyPI release. Add the feedstock update to the checklist.

ev-br · 2026-11-06T16:16:52Z

+1. **Check CI.** The release workflow runs its own tests, but it does not look at the other
+   workflows, so check that the latest commit on `main` is green.
+2. **Prepare the release in a pull request, and merge it.**
+    - Set `__version__` in `src/benchx/__init__.py`; `pyproject.toml` reads it from there.


dynamic versioning, ok :-).

ev-br · 2026-11-06T16:19:59Z

+   workflows, so check that the latest commit on `main` is green.
+2. **Prepare the release in a pull request, and merge it.**
+    - Set `__version__` in `src/benchx/__init__.py`; `pyproject.toml` reads it from there.
+    - Update `CHANGELOG.md`, including the list of contributors.


So how does the workflow work exactly: send the release PR, merge it, then push the tag? Or send the PR, push the tag, then merge the PR?
And when is the approval step from 1.?

As per releasing.md:

$ git fetch upstream $ git tag -a v0.1.0 -m "benchx 0.1.0" upstream/main $ git push upstream v0.1.0

The approval is through GitHub UI I believe.

ev-br · 2026-11-06T16:21:19Z

+
+4. **Approve the `publish-pypi` job** when the
+   [workflow run](https://github.com/benchx-dev/benchx/actions/workflows/release.yml) asks. Any
+   required reviewer of the `pypi` environment can approve it, including whoever pushed the tag.


Ah so the workflow itself asks for an approval, where? Is it somewhere in the Actions tab?

Yes, I believe here https://github.com/benchx-dev/benchx/actions/workflows/release.yml. I can't seem to be able to do it because I disabled selfrelease, see https://github.com/benchx-dev/benchx/settings/environments/26495556627/edit. Can you check if you see a release button?

responded on slack :-)

ev-br · 2026-11-06T16:22:58Z

+
+7. **Merge the conda-forge update.** Conda packages come from the benchx feedstock on
+   conda-forge, not from this repository. Once that feedstock exists, conda-forge's bot opens a
+   pull request on it after a new version reaches PyPI; review and merge that pull request.


Is conda-forge set up already, or do we need to set it up first?

Let's do that afterwards, opened #108

ev-br · 2026-11-06T16:25:36Z

+
+on:
+  push:
+    tags: ["v*"]


Agreed, there's no need to enforce specific versioning (v0.1.0 vs v2026.11.05 vs whatever other scheme)

Which shall we do thought? :)

0.1.0, 0.2.0, 0.2.1 (bugfix) etc sounds fine to me :-).

ev-br · 2026-11-06T16:26:57Z

+    branches: [main]
+    paths:
+      - ".github/workflows/release.yml"
+      - "pyproject.toml"


is this correct: pyproject.toml gets the version from src/benchx/__init__.py, which one triggers it?

ev-br · 2026-11-06T16:28:22Z

+      - uses: astral-sh/setup-uv@68559e99cc807c1eb7e9238fd43a15bba460eec2 # v10.3.3
+
+      - name: Build the sdist, and the wheel from it
+        run: uv build


TIL of an alternative to python -m build. If this works, it does, I suppose!

Let's see :D Haven't heard of it either.

https://docs.astral.sh/uv/guides/package/#building-your-package !

ev-br · 2026-11-06T16:38:28Z

+7. **Merge the conda-forge update.** Conda packages come from the benchx feedstock on
+   conda-forge, not from this repository. Once that feedstock exists, conda-forge's bot opens a
+   pull request on it after a new version reaches PyPI; review and merge that pull request.
+


I suppose the last step is to bump the version in benchx/src/__init__.py to the next .dev0: after releasing 0.1.0, the in-development version is 0.2.0.dev0 etc

Added 8 above

ev-br

Several comments ~~below~~above, all optional, and none is blocking.

…es in CI Start the next development version after a final release, so that results from main do not claim to come from the release. Say that a second person has to approve, since the pypi environment prevents self-review. Run the build job on pull requests that change the version.

rok added 6 commits November 6, 2026 16:52

MAINT: read the package version from __init__.py

69467df

The version was in both pyproject.toml and src/benchx/__init__.py. A release needs one value, since the tag, the packages and the version recorded in results must all agree.

MAINT: name the runner benchx, not benchx-prototype

e7ef2b3

The name is written into every result document, so change it before the first release.

DOC: add install instructions; make the UC-04 link absolute

42994d7

This page is the package's description on PyPI, where a relative link does not resolve.

DOC: add a release checklist

4189102

rok changed the title ~~CI: publish releases to PyPI and anaconda.org~~ CI: publish releases to PyPI Nov 6, 2026

rok mentioned this pull request Nov 6, 2026

Automate releases to PyPI #105

Closed

2 of 10 tasks

ev-br reviewed Nov 6, 2026

View reviewed changes

rok mentioned this pull request Nov 6, 2026

Distribute benchx through conda-forge #108

Open

2 of 5 tasks

ev-br reviewed Nov 6, 2026

View reviewed changes

ev-br previously approved these changes Nov 6, 2026

View reviewed changes

rok dismissed ev-br’s stale review via eac2c66 November 6, 2026 16:45

rok merged commit 0a69cbe into benchx-dev:main Nov 6, 2026
15 checks passed

Conversation

rok commented Nov 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes

Testing

Before the first release

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rok Nov 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rok Nov 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ev-br left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

rok commented Nov 6, 2026 •

edited

Loading

rok Nov 6, 2026 •

edited

Loading

rok Nov 6, 2026 •

edited

Loading

ev-br left a comment •

edited

Loading