Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 16 additions & 1 deletion .github/workflows/test_action.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,10 @@ concurrency:
jobs:
generate_data:
runs-on: ubuntu-latest
name: Run action on test file in repo
strategy:
matrix:
excluded_packages: ["", "numpy,\nscikit-learn"]
name: "Run action with exclusions: ${{ matrix.excluded_packages }}"
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
Expand All @@ -27,3 +30,15 @@ jobs:
project_file_name: tests/test_data/pyproject.toml
create_pr: false
schedule_path: tests/test_data/test_schedule.json
excluded_packages: ${{ matrix.excluded_packages }}
- name: Check dependency updates
env:
EXCLUDED_PACKAGES: ${{ matrix.excluded_packages }}
run: |
git diff -U0 -- tests/test_data/pyproject.toml > changes.diff
grep -q '^+.*pandas' changes.diff
if [ -n "$EXCLUDED_PACKAGES" ]; then
if grep -Eq '^[-+].*(numpy|scikit-learn)' changes.diff; then exit 1; fi
else
grep -q '^+.*numpy' changes.diff
fi
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,4 @@ schedule.json
__pycache__
*.pyc
*.lock
.DS_Store
6 changes: 3 additions & 3 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -15,19 +15,19 @@ repos:
- id: mixed-line-ending
- id: trailing-whitespace
- repo: https://github.com/rbubley/mirrors-prettier
rev: v3.8.3
rev: v3.9.6
hooks:
- id: prettier
files: \.(css|html|md|yml|yaml|gql)
args: [--prose-wrap=preserve]
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.15.14
rev: v0.16.6
hooks:
- id: ruff-check
args: ["--fix", "--show-fixes", "--exit-non-zero-on-fix"]
- id: ruff-format
- repo: https://github.com/codespell-project/codespell
rev: "v2.4.2"
rev: "v2.4.3"
hooks:
- id: codespell

Expand Down
7 changes: 6 additions & 1 deletion action.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,10 @@ inputs:
description: "If set, also update all non-SPEC0 dependencies to versions released within the last N years (e.g., 2)."
required: false
default: ""
excluded_packages:
description: "Comma- or whitespace-separated package names to leave unchanged, including with update_all. Use python to exclude Python requirements."
required: false
default: ""
runs:
using: "composite"
steps:
Expand Down Expand Up @@ -66,6 +70,7 @@ runs:
PROJECT_FILE_NAME: ${{ inputs.project_file_name }}
SCHEDULE_INPUT: ${{ inputs.schedule_path }}
UPDATE_ALL: ${{ inputs.update_all }}
EXCLUDED_PACKAGES: ${{ inputs.excluded_packages }}
run: |
set -e
if [ -n "$SCHEDULE_INPUT" ]; then
Expand All @@ -78,7 +83,7 @@ runs:
if [ -n "$UPDATE_ALL" ]; then
UPDATE_ALL_ARGS=(--update-all "$UPDATE_ALL")
fi
pixi run --manifest-path "${GITHUB_ACTION_PATH}/pyproject.toml" update-dependencies "${GITHUB_WORKSPACE}/${PROJECT_FILE_NAME}" "$SCHEDULE_PATH" "${UPDATE_ALL_ARGS[@]}"
pixi run --manifest-path "${GITHUB_ACTION_PATH}/pyproject.toml" update-dependencies "${GITHUB_WORKSPACE}/${PROJECT_FILE_NAME}" "$SCHEDULE_PATH" --excluded-packages "$EXCLUDED_PACKAGES" "${UPDATE_ALL_ARGS[@]}"
- name: Changes
id: changes
shell: bash
Expand Down
4 changes: 4 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -29,3 +29,7 @@ pytest = "*"

[tool.pixi.environments]
test = ["test"]

[tool.ruff.lint]
# Naive datetimes are fine: PyPI upload times are UTC and the schedule is quarter-granular
ignore = ["DTZ"]
15 changes: 15 additions & 0 deletions readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,11 +50,26 @@ The built-in `GITHUB_TOKEN` is used by default as long as the workflow has `pull
| `pr_title` | no | `chore: Drop support for unsupported packages conform SPEC 0` | Title of the opened PR |
| `commit_msg` | no | `chore: Drop support for unsupported packages conform SPEC 0` | Commit message for the version update commit |
| `update_all` | no | — | If set to a number N, also update non-SPEC0 dependencies to versions released within the last N years (e.g. `2`) |
| `excluded_packages` | no | — | Comma- or whitespace-separated package names to leave unchanged, including with `update_all` |

For examples of before/after see [tests/test_data/pyproject.toml](./tests/test_data/pyproject.toml) and [tests/test_data/pyproject_updated.toml](./tests/test_data/pyproject_updated.toml).

SPEC 0 packages include `ipython`, `matplotlib`, `networkx`, `numpy`, `pandas`, `scikit-image`, `scikit-learn`, `scipy`, `xarray`, and `zarr`.

### Excluding packages

To keep a package's lower bound as-is, for example to stay compatible with NumPy 1.26 while everything else updates, list it in `excluded_packages` before the action raises its bound (bounds are never lowered). Separate names with commas or whitespace; `python` excludes the Python requirement:

```yaml
with:
update_all: 2
excluded_packages: |
numpy, scikit-learn
python
```

Exclusions win over the schedule and `update_all`. The CLI takes the same value via `--excluded-packages`.

## Limitations

1. The action only tightens lower bounds and leaves upper bounds untouched. An update can produce an unsolvable environment — for example `numpy = ">=1.25.0,<2"` becomes `numpy = ">=2.0.0,<2"`. Keeping the environment solvable is out of scope; adjust upper bounds manually if needed.
Expand Down
17 changes: 14 additions & 3 deletions run_spec0_update.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
from spec0_action import update_pyproject_toml, read_toml, write_toml, read_schedule
from pathlib import Path
from argparse import ArgumentParser
from pathlib import Path

from spec0_action import read_schedule, read_toml, update_pyproject_toml, write_toml

if __name__ == "__main__":
parser = ArgumentParser(
Expand All @@ -24,6 +24,12 @@
metavar="YEARS",
help="Also update all non-SPEC0 dependencies to versions released within the last YEARS years (e.g., 2).",
)
parser.add_argument(
"--excluded-packages",
default="",
metavar="NAMES",
help="Leave these comma- or whitespace-separated package names unchanged; use python to exclude Python requirements.",
)
args = parser.parse_args()
toml_path = Path(args.toml_path)
schedule_path = Path(args.schedule_path)
Expand All @@ -37,5 +43,10 @@
)
project_data = read_toml(toml_path)
schedule_data = read_schedule(schedule_path)
update_pyproject_toml(project_data, schedule_data, update_all=args.update_all)
update_pyproject_toml(
project_data,
schedule_data,
update_all=args.update_all,
excluded_packages=args.excluded_packages.replace(",", " ").split(),
)
write_toml(toml_path, project_data)
66 changes: 35 additions & 31 deletions spec0_action/__init__.py
Original file line number Diff line number Diff line change
@@ -1,10 +1,18 @@
from functools import cache
from packaging.specifiers import SpecifierSet
from typing import Callable, Sequence, Dict
import datetime
from collections.abc import Callable, Sequence
from functools import cache

import requests
from packaging.specifiers import SpecifierSet
from packaging.utils import (
InvalidSdistFilename,
InvalidWheelFilename,
canonicalize_name,
parse_sdist_filename,
parse_wheel_filename,
)
from packaging.version import Version

from spec0_action.versions import repr_spec_set, tighten_lower_bound
from spec0_action.parsing import (
SupportSchedule,
Url,
Expand All @@ -15,24 +23,17 @@
read_toml,
write_toml,
)
from packaging.version import Version
from packaging.utils import (
InvalidSdistFilename,
InvalidWheelFilename,
canonicalize_name,
parse_sdist_filename,
parse_wheel_filename,
)
from spec0_action.versions import repr_spec_set, tighten_lower_bound

__all__ = ["read_schedule", "read_toml", "write_toml", "update_pyproject_toml"]
__all__ = ["read_schedule", "read_toml", "update_pyproject_toml", "write_toml"]


@cache
def _get_oldest_version_in_window(package: str, years: float) -> Version | None:
"""
Query PyPI, return oldest non-pre release version uploaded within the last ``years`` years.
"""
cutoff = datetime.datetime.now(tz=datetime.timezone.utc) - datetime.timedelta(
cutoff = datetime.datetime.now(tz=datetime.UTC) - datetime.timedelta(
days=int(365 * years)
)
try:
Expand All @@ -43,7 +44,7 @@ def _get_oldest_version_in_window(package: str, years: float) -> Version | None:
)
resp.raise_for_status()
data = resp.json()
except Exception:
except requests.RequestException:
return None
first_uploads: dict[Version, datetime.datetime] = {}
for f in data.get("files", []):
Expand Down Expand Up @@ -83,15 +84,15 @@ def _version_from_filename(filename: str) -> Version | None:
def update_pyproject_dependencies(
dependencies: list,
resolve_lower_bound: Callable[[str], Version | None],
own_name: str | None,
skip: set[str],
):
# Assign by index so the (tomlkit) list is updated in place
for i, dep_str in enumerate(dependencies):
if not isinstance(dep_str, str):
continue
pkg, extras, spec, env = parse_pep_dependency(dep_str)
package_key = canonicalize_name(pkg)
if isinstance(spec, Url) or package_key == own_name:
if isinstance(spec, Url) or package_key in skip:
continue
new_lower_bound = resolve_lower_bound(package_key)
if new_lower_bound is None:
Expand All @@ -117,11 +118,11 @@ def iter_pep_dependency_lists(pyproject_data: dict):


def update_dependency_table(
dep_table: dict, new_versions: Dict[str, Version], own_name: str | None
dep_table: dict, new_versions: dict[str, Version], skip: set[str]
):
for pkg, pkg_data in dep_table.items():
package_key = canonicalize_name(pkg)
if package_key == own_name or package_key not in new_versions:
if package_key in skip or package_key not in new_versions:
continue
# Like pkg = ">x.y.z,<a"
if isinstance(pkg_data, str):
Expand All @@ -145,12 +146,12 @@ def update_dependency_table(


def update_pixi_dependencies(
pixi_tables: dict, new_versions: Dict[str, Version], own_name: str | None
pixi_tables: dict, new_versions: dict[str, Version], skip: set[str]
):
for key in ("dependencies", "pypi-dependencies"):
dep_table = pixi_tables.get(key)
if isinstance(dep_table, dict):
update_dependency_table(dep_table, new_versions, own_name)
update_dependency_table(dep_table, new_versions, skip)

# Recurse into [tool.pixi.feature.X] and platform tables like
# [tool.pixi.target.linux-64], which hold the same dependency keys
Expand All @@ -159,7 +160,7 @@ def update_pixi_dependencies(
if isinstance(subtables, dict):
for subtable in subtables.values():
if isinstance(subtable, dict):
update_pixi_dependencies(subtable, new_versions, own_name)
update_pixi_dependencies(subtable, new_versions, skip)


def _update_requires_python(project_data: dict, new_lower_bound: Version):
Expand All @@ -182,6 +183,8 @@ def update_pyproject_toml(
pyproject_data: dict,
schedule_data: Sequence[SupportSchedule],
update_all: float | None = None,
*,
excluded_packages: Sequence[str] = (),
):
now = datetime.datetime.now(datetime.UTC)
applicable = sorted(
Expand All @@ -191,7 +194,7 @@ def update_pyproject_toml(
),
key=lambda s: datetime.datetime.fromisoformat(s["start_date"]),
)
new_version: Dict[str, Version] = {}
new_version: dict[str, Version] = {}
for schedule in applicable:
# Fill in the latest known requirement (schedule is sorted, newer entries overwrite older)
for pkg, version in schedule["packages"].items():
Expand All @@ -203,14 +206,15 @@ def update_pyproject_toml(
project_data = pyproject_data.get("project", {})
if not isinstance(project_data, dict):
project_data = {}
# Self-references like "pkg[extras]" are used to share extras between
# dependency groups, their version is always the local one so never pin it.
own_name = project_data.get("name")
own_name = canonicalize_name(own_name) if isinstance(own_name, str) else None

if "python" in new_version:
# Never touch excluded packages, nor self-references like "pkg[extras]" used
# to share extras between dependency groups (their version is always the local one).
skip = {canonicalize_name(pkg, validate=True) for pkg in excluded_packages}
if "python" in new_version and "python" not in skip:
_update_requires_python(project_data, new_version["python"])

if isinstance(own_name := project_data.get("name"), str):
skip.add(canonicalize_name(own_name))

def resolve_lower_bound(package_key: str) -> Version | None:
if package_key in new_version:
return new_version[package_key]
Expand All @@ -219,7 +223,7 @@ def resolve_lower_bound(package_key: str) -> Version | None:
return None

for dependencies in iter_pep_dependency_lists(pyproject_data):
update_pyproject_dependencies(dependencies, resolve_lower_bound, own_name)
update_pyproject_dependencies(dependencies, resolve_lower_bound, skip)

if "tool" in pyproject_data and "pixi" in pyproject_data["tool"]:
update_pixi_dependencies(pyproject_data["tool"]["pixi"], new_version, own_name)
update_pixi_dependencies(pyproject_data["tool"]["pixi"], new_version, skip)
21 changes: 11 additions & 10 deletions spec0_action/parsing.py
Original file line number Diff line number Diff line change
@@ -1,12 +1,13 @@
from typing import TypeAlias
from urllib.parse import ParseResult, urlparse
from tomlkit import dumps, loads
import json
from packaging.specifiers import InvalidSpecifier, SpecifierSet
from packaging.version import InvalidVersion, Version
from typing import Dict, Sequence, Tuple, TypedDict
from collections.abc import Sequence
from pathlib import Path
from re import compile
from typing import TypeAlias, TypedDict
from urllib.parse import ParseResult, urlparse

from packaging.specifiers import InvalidSpecifier, SpecifierSet
from packaging.version import InvalidVersion, Version
from tomlkit import dumps, loads

# We won't actually do anything with URLs we just need to detect them
Url: TypeAlias = ParseResult
Expand All @@ -19,7 +20,7 @@

class SupportSchedule(TypedDict):
start_date: str
packages: Dict[str, str]
packages: dict[str, str]


def parse_version_spec(s: str) -> SpecifierSet:
Expand Down Expand Up @@ -53,19 +54,19 @@ def write_toml(path: Path | str, data: dict):


def read_toml(path: Path | str) -> dict:
with open(path, "r") as file:
with open(path) as file:
contents = file.read()
return loads(contents)


def read_schedule(path: Path | str) -> Sequence[SupportSchedule]:
with open(path, "r") as file:
with open(path) as file:
return json.load(file)


def parse_pep_dependency(
dep_str: str,
) -> Tuple[str, str | None, SpecifierSet | Url | None, str | None]:
) -> tuple[str, str | None, SpecifierSet | Url | None, str | None]:
match = PEP_PACKAGE_IDENT_RE.match(dep_str)
if match is None:
raise ValueError("Could not find any valid python package identifier")
Expand Down
2 changes: 1 addition & 1 deletion spec0_action/versions.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
from packaging.version import Version
from packaging.specifiers import Specifier, SpecifierSet
from packaging.version import Version


def tighten_lower_bound(
Expand Down
Loading