diff --git a/docs/dependency-update-spec.md b/docs/dependency-update-spec.md new file mode 100644 index 00000000..d6b5ce01 --- /dev/null +++ b/docs/dependency-update-spec.md @@ -0,0 +1,196 @@ +# Spec: Dependency Range Update Utility + +Status: **draft** · Owner: TBD · Origin: manual process in PR #381 + +## 1. Goal + +Automate bumping the Python dependency **version ranges** in `pyproject.toml` so +they include the latest released versions, following the repo's existing +convention, keeping the PyPI (`[project]`) and conda (`[tool.pixi]`) specs in +sync, regenerating `pixi.lock`, validating with tests, and producing a +reviewable PR. + +This replaces the hand process used for PR #381 (`chore(deps): update dependency +ranges to include latest releases`). + +## 2. Why a script, and where Renovate fits + +We evaluated Renovate first. Findings (verified, not assumed): + +- Renovate's **pixi/conda manager *can* lift caps** (`>=0.119.1,<0.120 → + >=0.136.3,<0.137`) and regenerate `pixi.lock` in-branch via + `postUpgradeTasks`. Confirmed on a real Renovate PR. +- Renovate's **pep621/pep440 manager *cannot* rewrite a two-sided `>=x,=LATEST,=a,=latest,=a` (open) → **leave unchanged** by default (already includes latest); + `--bump-open-floors` raises the floor to `>=latest`. + - anything not starting with `>=` (`==`, `~=`, `3.12.*`, path/table deps, + unpinned) → **skip**. +4. Strip extras/markers when reading the name (`uvicorn[standard]` → `uvicorn`); + rewrite the version portion only. + +## 5. Version resolution + +For each in-scope package, determine the **target version**: + +| Package appears in | Source of "latest" | +|---|---| +| any `[tool.pixi.*]` table (conda) | conda-forge | +| both `[project]` and a conda table (synced) | **conda-forge** (binding for the lock); use for *both* specs | +| `[project]`/`[dependency-groups]` only (e.g. `x2s3`, `py-cluster-api`, `build`) | PyPI | + +Rules: +- Exclude pre-releases (unless `--allow-prerelease`). +- conda-forge frequently **lags** PyPI. Cap synced packages at the + conda-available version so the two specs stay in sync and `pixi.lock` + resolves; **report the divergence** (e.g. "uvicorn: PyPI 0.49.0 ahead of + conda-forge 0.48.0") so a human can decide whether to split the specs. +- Registry "latest" is a *proposal*; the authoritative check is whether + `pixi lock` can resolve it (§7). If lock fails for a package, back it off and + report. +- Name mapping: conda name may differ from PyPI name — support a `name_map` + override (rare here; watch `psycopg2-binary`). + +## 6. Algorithm + +1. Read `pyproject.toml` structure with `tomllib`; enumerate `(section, name, + current_spec)` and classify conda vs pypi-only vs synced. +2. Apply excludes/filters (`--exclude`, `--only`). +3. Resolve target versions (§5) with retries/backoff for flaky registries. +4. Compute new spec per §4; skip no-ops. +5. Rewrite `pyproject.toml` via **anchored, targeted regex** replacements + (preserve formatting/comments — do *not* round-trip through a TOML writer): + - `[project]`-style: `"name >=…"` (name + space) → avoids matching + `pydantic` vs `pydantic-settings`. + - pixi table key: `(?m)^name = "…"`. + - `[dependency-groups]` quoted no-space form: `"name>=…"`. + Sync `[project]` and `[tool.pixi.dependencies]`. +6. Regenerate `pixi.lock` (`pixi lock`); on failure, surface the conflict and + optionally retry without the offending bump. +7. Validate (§9). +8. Emit summary (§nine output). + +## 7. Lockfile / build considerations + +- `pixi.lock` **must** be regenerated after editing the manifest (any manifest + change invalidates `pixi install --locked`, which CI runs). +- The editable `fileglancer` pypi-dependency means `pixi` must **build** the + package to resolve PyPI deps. Therefore: + - Run in a **full checkout** (needs `frontend/package.json` for the + hatch-nodejs version hook); a stripped temp dir fails. + - `pixi lock --no-install` / `pixi upgrade --dry-run` **cannot** resolve (they + refuse to build the editable dep). Use a real `pixi lock`. +- `pixi` binary must be on `PATH`. + +## 8. CLI / configuration + +``` +pixi run python scripts/bump_deps.py [options] +``` + +Options: +- `--dry-run` (default) — print proposed changes; no writes. +- `--write` — apply to `pyproject.toml`. +- `--lock` — run `pixi lock` after writing. +- `--test` — run the backend test suite after locking. +- `--exclude PKG…` — default `{python, nodejs, pip}`. +- `--only PKG…` — restrict to these packages. +- `--bump-open-floors` — also raise floors of open `>=x` specs. +- `--allow-prerelease`. +- `--json` — machine-readable output (for CI / hybrid use). + +Optional config block (in `pyproject.toml`, e.g. `[tool.bump-deps]`): +```toml +[tool.bump-deps] +exclude = ["python", "nodejs", "pip"] +name_map = { } # pypi_name -> conda_name overrides +cap_policy = { fastapi = "minor", pandas = "major" } # per-pkg NEXT override +``` + +## 9. Validation & output + +- **Tests:** `pixi run -e test test-backend`. Note: suite collection currently + breaks on **stray untracked test files** (e.g. `tests/test_ssh_app.py` + importing a module not on the branch). Run on a **clean checkout** (git + worktree / stash) or pass `--ignore` for stray files. Frontend is unaffected + by Python bumps; leave to CI. +- **Output** (human): a table of `package | current → new | source | notes`, + plus sections for *skipped*, *already-latest*, and *PyPI-ahead-of-conda* + divergences, and prominent flagging of **major bumps**. + +## 10. PR workflow (optional `--pr`, or a thin wrapper) + +- Branch off `main`; commit **only** `pyproject.toml` + `pixi.lock`. +- Push to the fork over HTTPS using a token (SSH may be unavailable in CI). +- Use `--no-verify`: the Lefthook pre-push Prettier hook fails on unrelated + stray/working-tree files, and this change is Python-only. +- Open a (draft) PR to `JaneliaSciComp/fileglancer:main` with a body listing + runtime/dev bumps, notable majors, and conda-vs-PyPI divergences. + +## 11. Edge cases & risks + +- **Major bumps** (pandas 2→3, cachetools 6→7, pytest 8→9) may break code; + the test step is the guard — surface majors loudly. +- First-party packages (`x2s3`, `py-cluster-api`) are **PyPI-only** (not on + conda-forge). +- conda/PyPI **name** or **version** divergence (§5). +- Extras/markers in specs; environment markers. +- Never bump `python` / `requires-python`. +- Per-package cap policy (0ver vs semver) may need overriding. + +## 12. Non-goals + +- Resolving breaking-change fallout in source code. +- Frontend/npm updates. +- Replacing Renovate for non-pyproject ecosystems. + +## 13. Open questions + +1. Default for open `>=x` specs: leave (current default) or bump floor? +2. Should the script own PR creation, or only edit + lock and leave git to a + wrapper / CI? +3. Adopt the hybrid Renovate workflow (§2 option B) for scheduled runs, or keep + it on-demand? +4. Source of truth for "latest": registry query (fast, may propose + un-installable versions) vs. pixi-resolved (authoritative, heavier)? diff --git a/pyproject.toml b/pyproject.toml index f464b71f..cf480154 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -150,6 +150,7 @@ dev-launch = "pixi run uvicorn fileglancer.server:app --no-access-log --port 787 dev-launch-remote = "pixi run uvicorn fileglancer.server:app --host 0.0.0.0 --port 7878 --reload --ssl-keyfile /opt/certs/cert.key --ssl-certfile /opt/certs/cert.crt" prod-launch-remote = "pixi run uvicorn fileglancer.server:app --workers 10 --host 0.0.0.0 --port 7878 --ssl-keyfile /opt/certs/cert.key --ssl-certfile /opt/certs/cert.crt" dev-launch-secure = "python fileglancer/dev_launch.py" +bump-deps = "python scripts/bump_deps.py" migrate = "alembic -c fileglancer/alembic.ini upgrade head" migrate-create = "alembic -c fileglancer/alembic.ini revision --autogenerate" stamp-db = "python -m fileglancer.stamp_db" diff --git a/scripts/bump_deps.py b/scripts/bump_deps.py new file mode 100644 index 00000000..07b14fec --- /dev/null +++ b/scripts/bump_deps.py @@ -0,0 +1,708 @@ +#!/usr/bin/env python +"""Bump Python dependency *version ranges* in ``pyproject.toml``. + +Automates the manual process from PR #381: for each in-scope dependency, look up +the latest released version and rewrite its range so it includes that version, +following the repo convention ``>=LATEST, ``pixi search -c conda-forge`` (binding for the lock); + * PyPI-only packages (e.g. ``x2s3``, ``py-cluster-api``, ``build``) -> PyPI JSON. + +This script only edits the manifest, locks, and tests. Git/PR creation is left to +a wrapper or CI (spec §10 option A). +""" + +from __future__ import annotations + +import json +import re +import subprocess +import sys +import tomllib +import urllib.error +import urllib.request +from concurrent.futures import ThreadPoolExecutor +from dataclasses import dataclass, field +from pathlib import Path +from typing import Optional + +import click +from loguru import logger +from packaging.version import InvalidVersion, Version + +REPO_ROOT = Path(__file__).resolve().parent.parent +PYPROJECT = REPO_ROOT / "pyproject.toml" + +# Toolchain pins we never touch (spec §3). Extendable via [tool.bump-deps].exclude. +DEFAULT_EXCLUDE = {"python", "nodejs", "pip"} + +PYPI_JSON_URL = "https://pypi.org/pypi/{name}/json" + +# Splits a PEP 508 requirement string into (name[extras], rest). ``rest`` holds +# the version specifier and any environment marker. +_REQ_RE = re.compile(r"^\s*([A-Za-z0-9][A-Za-z0-9._-]*)\s*(\[[^\]]*\])?\s*(.*)$") + + +# --------------------------------------------------------------------------- # +# Pure helpers (no I/O — unit tested) +# --------------------------------------------------------------------------- # +@dataclass +class SpecInfo: + """Classification of a single version specifier string.""" + + kind: str # "capped" | "open" | "skip" + floor: Optional[str] = None # version after ">=" + cap: Optional[str] = None # version after "<" + raw: str = "" + + +def split_requirement(entry: str) -> tuple[str, str]: + """Split a PEP 508 string into (canonical_name, version_spec). + + Strips extras and environment markers; returns the version portion only. + + "uvicorn[standard] >=0.48.0,<0.49 ; python_version>='3.9'" + -> ("uvicorn", ">=0.48.0,<0.49") + """ + m = _REQ_RE.match(entry) + if not m: + return entry.strip(), "" + name = m.group(1) + rest = m.group(3) or "" + # Drop environment markers. + spec = rest.split(";", 1)[0].strip() + return name, spec + + +def parse_spec(spec: str) -> SpecInfo: + """Classify a version specifier per spec §4.3. + + ``>=a, capped; ``>=a`` -> open; anything else (``==``, ``~=``, + ``3.12.*``, table/path deps, unpinned) -> skip. + """ + s = spec.strip() + if not s.startswith(">="): + return SpecInfo(kind="skip", raw=spec) + parts = [p.strip() for p in s.split(",")] + floor = parts[0][2:].strip() + if not floor: + return SpecInfo(kind="skip", raw=spec) + if len(parts) == 1: + return SpecInfo(kind="open", floor=floor, raw=spec) + if len(parts) == 2 and parts[1].startswith("<") and not parts[1].startswith("<="): + cap = parts[1][1:].strip() + return SpecInfo(kind="capped", floor=floor, cap=cap, raw=spec) + # Multiple constraints or "<=" cap: leave it alone. + return SpecInfo(kind="skip", raw=spec) + + +def compute_next(latest: Version, policy: Optional[str] = None) -> str: + """Compute the exclusive upper bound NEXT for a target version (spec §4.1). + + Default: 0ver-aware — ``0.x`` bumps the minor, otherwise bump the major. + ``policy`` ("minor"/"major") overrides per-package via [tool.bump-deps]. + """ + if policy == "major": + return str(latest.major + 1) + if policy == "minor": + if latest.major == 0: + return f"0.{latest.minor + 1}" + return f"{latest.major}.{latest.minor + 1}" + # Default convention. + if latest.major == 0: + return f"0.{latest.minor + 1}" + return str(latest.major + 1) + + +def compute_new_spec( + info: SpecInfo, + latest: Version, + policy: Optional[str] = None, + bump_open_floors: bool = False, +) -> Optional[str]: + """Compute the new specifier string, or ``None`` for a no-op / skip. + + Preserves each spec's shape (spec §4.3): + * ``>=a, ``>=latest,=a`` -> unchanged, unless ``bump_open_floors`` -> ``>=latest`` + * skip -> ``None`` + """ + if info.kind == "skip": + return None + if info.kind == "open": + if not bump_open_floors: + return None + new = f">={latest}" + return new if new != info.raw.strip() else None + # capped + next_cap = compute_next(latest, policy) + new = f">={latest},<{next_cap}" + return new if new != info.raw.strip() else None + + +# --------------------------------------------------------------------------- # +# Enumeration +# --------------------------------------------------------------------------- # +@dataclass +class DepRef: + """A single occurrence of a dependency in some manifest table.""" + + name: str + spec: str + table: str # human label, e.g. "project", "pixi", "pixi.feature.test" + form: str # "project" | "pixi_key" | "group_quoted" + + +def _project_deps(doc: dict) -> list[tuple[str, str]]: + return list(doc.get("project", {}).get("dependencies", []) or []) + + +def enumerate_deps(doc: dict) -> list[DepRef]: + """Walk the in-scope tables (spec §3) yielding a DepRef per occurrence.""" + refs: list[DepRef] = [] + + # [project].dependencies (PEP 508 strings, "name >=...") + for entry in _project_deps(doc): + name, spec = split_requirement(entry) + refs.append(DepRef(name=name, spec=spec, table="project", form="project")) + + pixi = doc.get("tool", {}).get("pixi", {}) + + # [tool.pixi.dependencies] + for name, spec in (pixi.get("dependencies", {}) or {}).items(): + if isinstance(spec, str): + refs.append(DepRef(name=name, spec=spec, table="pixi", form="pixi_key")) + + # [tool.pixi.feature.*.dependencies] + for feat_name, feat in (pixi.get("feature", {}) or {}).items(): + for name, spec in (feat.get("dependencies", {}) or {}).items(): + if isinstance(spec, str): + refs.append( + DepRef( + name=name, + spec=spec, + table=f"pixi.feature.{feat_name}", + form="pixi_key", + ) + ) + + # [dependency-groups].* (PEP 508 strings, often "name>=..." no space) + for group_name, members in (doc.get("dependency-groups", {}) or {}).items(): + for entry in members or []: + if not isinstance(entry, str): + continue # skip {include-group = ...} tables + name, spec = split_requirement(entry) + refs.append( + DepRef( + name=name, + spec=spec, + table=f"group.{group_name}", + form="group_quoted", + ) + ) + + return refs + + +def classify_sources(refs: list[DepRef], doc: dict) -> dict[str, str]: + """Map each package name to a source category: synced | conda | pypi_only.""" + project_names = {split_requirement(e)[0] for e in _project_deps(doc)} + pixi_main = set((doc.get("tool", {}).get("pixi", {}).get("dependencies", {}) or {})) + + conda_names: set[str] = set() + for r in refs: + if r.form == "pixi_key": + conda_names.add(r.name) + + categories: dict[str, str] = {} + for r in refs: + if r.name in categories: + continue + if r.name in project_names and r.name in pixi_main: + categories[r.name] = "synced" + elif r.name in conda_names: + categories[r.name] = "conda" + else: + categories[r.name] = "pypi_only" + return categories + + +# --------------------------------------------------------------------------- # +# Version resolution (network — mocked in tests) +# --------------------------------------------------------------------------- # +def _max_version(versions: list[Version], allow_prerelease: bool) -> Optional[Version]: + candidates = [v for v in versions if allow_prerelease or not v.is_prerelease] + return max(candidates) if candidates else None + + +def conda_latest(name: str, allow_prerelease: bool = False) -> Optional[Version]: + """Latest conda-forge version via ``pixi search`` (spec §5).""" + try: + proc = subprocess.run( + ["pixi", "search", name, "-c", "conda-forge"], + capture_output=True, + text=True, + cwd=REPO_ROOT, + timeout=120, + ) + except (FileNotFoundError, subprocess.TimeoutExpired) as exc: + logger.warning("pixi search failed for {}: {}", name, exc) + return None + if proc.returncode != 0: + logger.debug("pixi search non-zero for {}: {}", name, proc.stderr.strip()) + return None + + versions: list[Version] = [] + for line in proc.stdout.splitlines(): + stripped = line.strip() + # The matched (newest) block has a "Version X.Y.Z" field; the + # "Other Versions" table lists older ones in the first column. + if stripped.startswith("Version"): + tokens = stripped.split() + if len(tokens) >= 2: + try: + versions.append(Version(tokens[1])) + except InvalidVersion: + pass + else: + tok = stripped.split() + if tok: + try: + versions.append(Version(tok[0])) + except InvalidVersion: + pass + return _max_version(versions, allow_prerelease) + + +def pypi_latest(name: str, allow_prerelease: bool = False) -> Optional[Version]: + """Latest PyPI version from the JSON API, skipping fully-yanked releases.""" + url = PYPI_JSON_URL.format(name=name) + try: + with urllib.request.urlopen(url, timeout=30) as resp: + data = json.load(resp) + except (urllib.error.URLError, urllib.error.HTTPError, TimeoutError) as exc: + logger.warning("PyPI lookup failed for {}: {}", name, exc) + return None + + versions: list[Version] = [] + for ver, files in (data.get("releases", {}) or {}).items(): + if files and all(f.get("yanked") for f in files): + continue # entirely yanked + try: + versions.append(Version(ver)) + except InvalidVersion: + continue + return _max_version(versions, allow_prerelease) + + +@dataclass +class Target: + """Resolved target version for a package.""" + + name: str + category: str + version: Optional[Version] + source: str # "conda-forge" | "PyPI" | "unresolved" + note: str = "" + + +def resolve_target( + name: str, + category: str, + allow_prerelease: bool, + name_map: dict[str, str], +) -> Target: + """Resolve the binding target version + report conda/PyPI divergence (spec §5).""" + if category in ("synced", "conda"): + conda_name = name_map.get(name, name) + cv = conda_latest(conda_name, allow_prerelease) + note = "" + if category == "synced": + pv = pypi_latest(name, allow_prerelease) + if pv and cv and pv > cv: + note = f"PyPI {pv} ahead of conda-forge {cv}" + if cv is None: + return Target(name, category, None, "unresolved", "not found on conda-forge") + return Target(name, category, cv, "conda-forge", note) + + # pypi_only + pv = pypi_latest(name, allow_prerelease) + if pv is None: + return Target(name, category, None, "unresolved", "not found on PyPI") + return Target(name, category, pv, "PyPI", "") + + +# --------------------------------------------------------------------------- # +# Rewrite (targeted, formatting-preserving — spec §6.5) +# --------------------------------------------------------------------------- # +@dataclass +class Change: + name: str + old_spec: str + new_spec: str + source: str + tables: list[str] = field(default_factory=list) + is_major: bool = False + note: str = "" + + +def _rewrite_one(text: str, ref: DepRef, new_spec: str) -> str: + """Apply a single spec rewrite for ``ref`` to the raw manifest text. + + Uses anchored matches keyed on the package name so e.g. ``pydantic`` never + matches inside ``pydantic-settings``. Asserts exactly one substitution. + """ + if ref.form == "pixi_key": + # ^name = ">=..." + pattern = re.compile( + r'(?m)^(\s*' + re.escape(ref.name) + r'\s*=\s*")' + + re.escape(ref.spec) + + r'(")' + ) + repl = r"\g<1>" + new_spec.replace("\\", "\\\\") + r"\g<2>" + elif ref.form == "project": + # "name " (name followed by whitespace, inside quotes) + pattern = re.compile( + r'("' + re.escape(ref.name) + r'(?:\[[^\]]*\])?\s+)' + + re.escape(ref.spec) + ) + repl = r"\g<1>" + new_spec.replace("\\", "\\\\") + else: # group_quoted: "name>=..." (no space) + pattern = re.compile( + r'("' + re.escape(ref.name) + r'(?:\[[^\]]*\])?)' + + re.escape(ref.spec) + ) + repl = r"\g<1>" + new_spec.replace("\\", "\\\\") + + new_text, n = pattern.subn(repl, text) + if n != 1: + raise RuntimeError( + f"expected exactly 1 rewrite for {ref.name} in {ref.table} " + f"({ref.form}), found {n}" + ) + return new_text + + +def apply_changes(text: str, refs: list[DepRef], new_specs: dict[str, str]) -> str: + """Rewrite every in-scope ref whose package has a new spec.""" + for ref in refs: + new_spec = new_specs.get(ref.name) + if new_spec is None: + continue + # Only rewrite refs whose current spec is itself capped/open and differs. + info = parse_spec(ref.spec) + if info.kind == "skip" or new_spec == ref.spec.strip(): + continue + text = _rewrite_one(text, ref, new_spec) + return text + + +# --------------------------------------------------------------------------- # +# Config +# --------------------------------------------------------------------------- # +@dataclass +class Config: + exclude: set[str] + name_map: dict[str, str] + cap_policy: dict[str, str] + + +def load_config(doc: dict) -> Config: + block = doc.get("tool", {}).get("bump-deps", {}) or {} + exclude = set(DEFAULT_EXCLUDE) | set(block.get("exclude", []) or []) + name_map = dict(block.get("name_map", {}) or {}) + cap_policy = dict(block.get("cap_policy", {}) or {}) + return Config(exclude=exclude, name_map=name_map, cap_policy=cap_policy) + + +# --------------------------------------------------------------------------- # +# Orchestration helpers +# --------------------------------------------------------------------------- # +def plan_changes( + doc: dict, + cfg: Config, + only: set[str], + extra_exclude: set[str], + allow_prerelease: bool, + bump_open_floors: bool, +) -> tuple[list[Change], list[DepRef], dict[str, str], list[tuple[str, str]]]: + """Resolve targets and compute the set of changes. + + Returns ``(changes, refs, new_specs, skipped)`` where ``skipped`` is a list + of ``(name, reason)`` for reporting. + """ + refs = enumerate_deps(doc) + categories = classify_sources(refs, doc) + + exclude = cfg.exclude | extra_exclude + names = sorted({r.name for r in refs}) + selected = [ + n for n in names if n not in exclude and (not only or n in only) + ] + + # Resolve targets in parallel (each conda lookup spawns a pixi subprocess). + targets: dict[str, Target] = {} + with ThreadPoolExecutor(max_workers=8) as pool: + futs = { + pool.submit( + resolve_target, n, categories[n], allow_prerelease, cfg.name_map + ): n + for n in selected + } + for fut in futs: + t = fut.result() + targets[t.name] = t + + changes: list[Change] = [] + new_specs: dict[str, str] = {} + skipped: list[tuple[str, str]] = [] + + # Group refs by name to compute one new spec per package (shapes match across + # tables in practice; we use the [project]/pixi ref as the shape reference). + by_name: dict[str, list[DepRef]] = {} + for r in refs: + by_name.setdefault(r.name, []).append(r) + + for name in selected: + target = targets.get(name) + pkg_refs = by_name[name] + if target is None or target.version is None: + skipped.append((name, target.note if target else "unresolved")) + continue + + # Pick a representative ref for shape (prefer a capped one). + infos = [(r, parse_spec(r.spec)) for r in pkg_refs] + capped = [(r, i) for r, i in infos if i.kind == "capped"] + open_ = [(r, i) for r, i in infos if i.kind == "open"] + if capped: + ref, info = capped[0] + elif open_: + ref, info = open_[0] + else: + skipped.append((name, f"unsupported spec ({pkg_refs[0].spec!r})")) + continue + + policy = cfg.cap_policy.get(name) + try: + floor_v = Version(info.floor) if info.floor else None + except InvalidVersion: + floor_v = None + if floor_v is not None and target.version < floor_v: + skipped.append( + (name, f"registry {target.version} behind current floor {floor_v}") + ) + continue + + new_spec = compute_new_spec(info, target.version, policy, bump_open_floors) + if new_spec is None: + continue # already latest / open floor left as-is + new_specs[name] = new_spec + + is_major = info.kind == "capped" and info.cap is not None and ( + _is_major_bump(info.cap, target.version) + ) + changes.append( + Change( + name=name, + old_spec=info.raw.strip(), + new_spec=new_spec, + source=target.source, + tables=sorted({r.table for r in pkg_refs}), + is_major=is_major, + note=target.note, + ) + ) + + return changes, refs, new_specs, skipped + + +def _is_major_bump(old_cap: str, latest: Version) -> bool: + """Heuristic: does ``latest`` cross the previous upper bound's major?""" + try: + cap_v = Version(old_cap) + except InvalidVersion: + return False + # old cap like "8" means previous major was 7; latest.major >= cap implies bump. + if latest.major == 0: + return False + return latest.major >= cap_v.major + + +# --------------------------------------------------------------------------- # +# Output +# --------------------------------------------------------------------------- # +def render_human( + changes: list[Change], + skipped: list[tuple[str, str]], + divergences: list[Change], + selected_count: int, +) -> str: + lines: list[str] = [] + lines.append("") + if changes: + lines.append(f"Proposed changes ({len(changes)}):") + width = max(len(c.name) for c in changes) + for c in changes: + flag = " ⚠ MAJOR" if c.is_major else "" + note = f" [{c.note}]" if c.note else "" + lines.append( + f" {c.name:<{width}} {c.old_spec} → {c.new_spec}" + f" ({c.source}){flag}{note}" + ) + else: + lines.append("No changes proposed — all in-scope specs already include the latest.") + + majors = [c for c in changes if c.is_major] + if majors: + lines.append("") + lines.append(f"⚠ MAJOR bumps ({len(majors)}) — review for breaking changes:") + for c in majors: + lines.append(f" {c.name}: {c.old_spec} → {c.new_spec}") + + if divergences: + lines.append("") + lines.append("PyPI ahead of conda-forge (specs kept in sync at conda version):") + for c in divergences: + lines.append(f" {c.name}: {c.note}") + + if skipped: + lines.append("") + lines.append(f"Skipped ({len(skipped)}):") + for name, reason in skipped: + lines.append(f" {name}: {reason}") + + return "\n".join(lines) + + +def render_json( + changes: list[Change], + skipped: list[tuple[str, str]], +) -> str: + payload = { + "changes": [ + { + "name": c.name, + "old": c.old_spec, + "new": c.new_spec, + "source": c.source, + "tables": c.tables, + "major": c.is_major, + "note": c.note, + } + for c in changes + ], + "skipped": [{"name": n, "reason": r} for n, r in skipped], + } + return json.dumps(payload, indent=2) + + +# --------------------------------------------------------------------------- # +# Subprocess steps +# --------------------------------------------------------------------------- # +def run_lock() -> bool: + """Regenerate pixi.lock with a real ``pixi lock`` (spec §7).""" + logger.info("Running 'pixi lock' ...") + proc = subprocess.run(["pixi", "lock"], cwd=REPO_ROOT) + if proc.returncode != 0: + logger.error( + "pixi lock failed — a bump likely conflicts. Re-run with --exclude on " + "the offending package, or split its [project]/[tool.pixi] specs." + ) + return False + return True + + +def run_tests() -> bool: + """Run the backend test suite (spec §9). + + Note: collection can break on stray untracked test files; run on a clean + checkout or remove/ignore them first. + """ + logger.info("Running backend tests ('pixi run -e test test-backend') ...") + proc = subprocess.run( + ["pixi", "run", "-e", "test", "test-backend"], cwd=REPO_ROOT + ) + return proc.returncode == 0 + + +# --------------------------------------------------------------------------- # +# CLI +# --------------------------------------------------------------------------- # +@click.command() +@click.option("--write", is_flag=True, help="Apply changes to pyproject.toml (default: dry-run).") +@click.option("--dry-run", is_flag=True, help="Force dry-run even if --write is given.") +@click.option("--lock", is_flag=True, help="Run 'pixi lock' after writing.") +@click.option("--test", "run_test", is_flag=True, help="Run the backend test suite after locking.") +@click.option("--exclude", multiple=True, help="Extra package(s) to skip (adds to defaults).") +@click.option("--only", multiple=True, help="Restrict to these package(s).") +@click.option("--bump-open-floors", is_flag=True, help="Raise floors of open '>=x' specs to '>=latest'.") +@click.option("--allow-prerelease", is_flag=True, help="Consider pre-release versions.") +@click.option("--json", "as_json", is_flag=True, help="Machine-readable JSON output.") +def main( + write: bool, + dry_run: bool, + lock: bool, + run_test: bool, + exclude: tuple[str, ...], + only: tuple[str, ...], + bump_open_floors: bool, + allow_prerelease: bool, + as_json: bool, +) -> None: + """Bump dependency version ranges in pyproject.toml to include latest releases.""" + write_mode = write and not dry_run + + raw = PYPROJECT.read_text() + doc = tomllib.loads(raw) + cfg = load_config(doc) + + changes, refs, new_specs, skipped = plan_changes( + doc=doc, + cfg=cfg, + only=set(only), + extra_exclude=set(exclude), + allow_prerelease=allow_prerelease, + bump_open_floors=bump_open_floors, + ) + divergences = [c for c in changes if c.note] + + if as_json: + click.echo(render_json(changes, skipped)) + else: + selected_count = len({r.name for r in refs}) + click.echo(render_human(changes, skipped, divergences, selected_count)) + + if not changes: + return + + if not write_mode: + if not as_json: + click.echo("\n(dry-run — pass --write to apply)") + return + + new_text = apply_changes(raw, refs, new_specs) + PYPROJECT.write_text(new_text) + logger.info("Wrote {} change(s) to {}", len(changes), PYPROJECT) + + if lock: + if not run_lock(): + sys.exit(1) + elif not as_json: + click.echo("\n(remember to run 'pixi lock' — pass --lock to do it now)") + + if run_test: + if not lock: + logger.warning("--test without --lock: testing against a stale lockfile.") + if not run_tests(): + logger.error("Backend tests failed.") + sys.exit(1) + + +if __name__ == "__main__": + main() diff --git a/tests/test_bump_deps.py b/tests/test_bump_deps.py new file mode 100644 index 00000000..873a55de --- /dev/null +++ b/tests/test_bump_deps.py @@ -0,0 +1,249 @@ +"""Unit tests for scripts/bump_deps.py (pure logic; network/pixi mocked).""" + +import sys +import tomllib +from pathlib import Path + +import pytest +from packaging.version import Version + +# scripts/ is not a package — make it importable. +sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "scripts")) + +import bump_deps as bd # noqa: E402 + + +# --------------------------------------------------------------------------- # +# split_requirement +# --------------------------------------------------------------------------- # +def test_split_requirement_basic(): + """Name and spec split on whitespace.""" + assert bd.split_requirement("alembic >=1.18.4,<2") == ("alembic", ">=1.18.4,<2") + + +def test_split_requirement_no_space(): + """[dependency-groups] no-space form.""" + assert bd.split_requirement("build>=1.5.0,<2") == ("build", ">=1.5.0,<2") + + +def test_split_requirement_strips_extras(): + """Extras are dropped from the name (spec §4.4).""" + assert bd.split_requirement("uvicorn[standard] >=0.48.0,<0.49") == ( + "uvicorn", + ">=0.48.0,<0.49", + ) + + +def test_split_requirement_strips_markers(): + """Environment markers are dropped from the spec.""" + name, spec = bd.split_requirement("foo >=1.0,<2 ; python_version>='3.9'") + assert name == "foo" + assert spec == ">=1.0,<2" + + +# --------------------------------------------------------------------------- # +# parse_spec +# --------------------------------------------------------------------------- # +def test_parse_spec_capped(): + info = bd.parse_spec(">=1.18.4,<2") + assert info.kind == "capped" + assert info.floor == "1.18.4" + assert info.cap == "2" + + +def test_parse_spec_open(): + info = bd.parse_spec(">=24.0") + assert info.kind == "open" + assert info.floor == "24.0" + assert info.cap is None + + +@pytest.mark.parametrize( + "spec", + ["==1.2.3", "~=2.0", "3.12.*", "1.2.3", "", ">=1,<=2", ">=1,<2,<3"], +) +def test_parse_spec_skip(spec): + """Anything not a clean >=a or >=a, next minor + ("0.136.3", "0.137"), + ("1.18.4", "2"), # semver -> next major + ("7.1.4", "8"), + ("3.0.3", "4"), + ], +) +def test_compute_next_default(version, expected): + assert bd.compute_next(Version(version)) == expected + + +def test_compute_next_policy_override(): + """cap_policy can force minor/major regardless of 0ver.""" + assert bd.compute_next(Version("2.13.4"), policy="minor") == "2.14" + assert bd.compute_next(Version("0.48.0"), policy="major") == "1" + + +# --------------------------------------------------------------------------- # +# compute_new_spec +# --------------------------------------------------------------------------- # +def test_compute_new_spec_capped(): + info = bd.parse_spec(">=0.119.1,<0.120") + assert bd.compute_new_spec(info, Version("0.136.3")) == ">=0.136.3,<0.137" + + +def test_compute_new_spec_capped_semver(): + info = bd.parse_spec(">=6.2.1,<7") + assert bd.compute_new_spec(info, Version("7.1.4")) == ">=7.1.4,<8" + + +def test_compute_new_spec_noop(): + """Already-latest spec yields no change.""" + info = bd.parse_spec(">=0.136.3,<0.137") + assert bd.compute_new_spec(info, Version("0.136.3")) is None + + +def test_compute_new_spec_open_left_alone(): + """Open specs are unchanged by default.""" + info = bd.parse_spec(">=24.0") + assert bd.compute_new_spec(info, Version("25.1")) is None + + +def test_compute_new_spec_open_bump_floor(): + """--bump-open-floors raises the floor but keeps it open.""" + info = bd.parse_spec(">=24.0") + assert bd.compute_new_spec(info, Version("25.1"), bump_open_floors=True) == ">=25.1" + + +def test_compute_new_spec_skip(): + info = bd.parse_spec("==1.2.3") + assert bd.compute_new_spec(info, Version("2.0.0")) is None + + +# --------------------------------------------------------------------------- # +# Rewrite +# --------------------------------------------------------------------------- # +SAMPLE = """\ +[project] +dependencies = [ + "pydantic >=2.10.6,<3", + "pydantic-settings >=2.11.0,<3", + "uvicorn[standard] >=0.38.0,<0.39", + "packaging >=24.0", +] + +[tool.pixi.dependencies] +pydantic = ">=2.10.6,<3" +pydantic-settings = ">=2.11.0,<3" + +[dependency-groups] +release = ["build>=1.5.0,<2"] +""" + + +def _doc(): + return tomllib.loads(SAMPLE) + + +def test_rewrite_project_form_disambiguates_prefix(): + """Rewriting pydantic must not touch pydantic-settings.""" + doc = _doc() + refs = bd.enumerate_deps(doc) + new_specs = {"pydantic": ">=2.13.4,<3"} + out = bd.apply_changes(SAMPLE, refs, new_specs) + assert '"pydantic >=2.13.4,<3"' in out + assert '"pydantic-settings >=2.11.0,<3"' in out # untouched + assert 'pydantic = ">=2.13.4,<3"' in out # pixi key synced + assert 'pydantic-settings = ">=2.11.0,<3"' in out + + +def test_rewrite_extras_preserved(): + """Extras stay in the project string after a version rewrite.""" + doc = _doc() + refs = bd.enumerate_deps(doc) + out = bd.apply_changes(SAMPLE, refs, {"uvicorn": ">=0.48.0,<0.49"}) + assert '"uvicorn[standard] >=0.48.0,<0.49"' in out + + +def test_rewrite_group_quoted_form(): + """No-space [dependency-groups] form is rewritten in place.""" + doc = _doc() + refs = bd.enumerate_deps(doc) + out = bd.apply_changes(SAMPLE, refs, {"build": ">=1.6.0,<2"}) + assert '"build>=1.6.0,<2"' in out + + +def test_rewrite_idempotent(): + """Applying the same target twice is a no-op the second time.""" + doc = _doc() + refs = bd.enumerate_deps(doc) + new_specs = {"pydantic": ">=2.13.4,<3"} + once = bd.apply_changes(SAMPLE, refs, new_specs) + twice = bd.apply_changes(once, bd.enumerate_deps(tomllib.loads(once)), new_specs) + assert once == twice + + +# --------------------------------------------------------------------------- # +# enumerate_deps / classify_sources +# --------------------------------------------------------------------------- # +def test_enumerate_and_classify(): + doc = _doc() + refs = bd.enumerate_deps(doc) + names = {r.name for r in refs} + assert {"pydantic", "pydantic-settings", "uvicorn", "packaging", "build"} <= names + + cats = bd.classify_sources(refs, doc) + assert cats["pydantic"] == "synced" # in both [project] and [tool.pixi] + assert cats["uvicorn"] == "pypi_only" # only in [project] here + assert cats["build"] == "pypi_only" # [dependency-groups] + + +# --------------------------------------------------------------------------- # +# resolve_target (network mocked) +# --------------------------------------------------------------------------- # +def test_resolve_target_synced_uses_conda(monkeypatch): + monkeypatch.setattr(bd, "conda_latest", lambda n, allow=False: Version("2.13.4")) + monkeypatch.setattr(bd, "pypi_latest", lambda n, allow=False: Version("2.13.4")) + t = bd.resolve_target("pydantic", "synced", False, {}) + assert t.source == "conda-forge" + assert t.version == Version("2.13.4") + assert t.note == "" + + +def test_resolve_target_reports_pypi_ahead(monkeypatch): + """Synced package caps at conda version and notes the divergence (spec §5).""" + monkeypatch.setattr(bd, "conda_latest", lambda n, allow=False: Version("0.48.0")) + monkeypatch.setattr(bd, "pypi_latest", lambda n, allow=False: Version("0.49.0")) + t = bd.resolve_target("uvicorn", "synced", False, {}) + assert t.version == Version("0.48.0") + assert "ahead of conda-forge" in t.note + + +def test_resolve_target_pypi_only(monkeypatch): + monkeypatch.setattr(bd, "pypi_latest", lambda n, allow=False: Version("1.3.0")) + t = bd.resolve_target("x2s3", "pypi_only", False, {}) + assert t.source == "PyPI" + assert t.version == Version("1.3.0") + + +def test_resolve_target_unresolved(monkeypatch): + monkeypatch.setattr(bd, "conda_latest", lambda n, allow=False: None) + monkeypatch.setattr(bd, "pypi_latest", lambda n, allow=False: None) + t = bd.resolve_target("ghost", "conda", False, {}) + assert t.version is None + assert t.source == "unresolved" + + +# --------------------------------------------------------------------------- # +# pypi_latest version selection +# --------------------------------------------------------------------------- # +def test_max_version_excludes_prerelease(): + versions = [Version("1.0.0"), Version("2.0.0rc1"), Version("1.5.0")] + assert bd._max_version(versions, allow_prerelease=False) == Version("1.5.0") + assert bd._max_version(versions, allow_prerelease=True) == Version("2.0.0rc1")