Skip to content

fix(dependencies): read pyproject.toml as TOML and mark dev declarations - #134

Merged
matiasdaloia merged 1 commit into
mainfrom
fix/pyproject-dependency-scope
Nov 6, 2026
Merged

matiasdaloia merged 1 commit into
mainfrom
fix/pyproject-dependency-scope

Conversation

@matiasdaloia

Copy link
Copy Markdown
Contributor

Why

A self-hosted Earnie customer scanned a Python project with no lockfile and got its dev tools (pytest, pylint) back as ordinary runtime dependencies, so a policy that skips dev dependencies flagged pylint as GPL. The manifest parser gave no way to tell a dev declaration from a runtime one, and its pyproject.toml reader also lost or invented dependencies.

What was broken

ParsePyprojectToml matched lines: any line holding dependencies and [ opened a "section". On the customer's file

[project]
name = "earnie-staging-probe"
version = "0.1.0"
dependencies = ["requests>=2.31"]
[project.optional-dependencies]
dev = ["pytest>=7.4", "pylint>=2.17"]

it returned pytest and pylint with no scope and dropped requests entirely. The new tests run against the old parser (v0.16.0) show the rest of the damage:

project_dependencies_and_a_dev_extra: got 2 purls, want 3 (requests missing, pytest/pylint Scope "")
other_project_strings...: got 3 purls, want 2 (+ pkg:pypi/README from readme = "README.md")
poetry_dependencies...: got 2 purls, want 6 ([pkg:pypi/2 pkg:pypi/23] from flask = {version = "2.3.0"} and black = "23.1")
every_PEP_735_group_is_dev...: got 0 purls, want 3
an_extra_is_dev_only_when_its_name_says_so: got 0 purls, want 3
TestParsePyprojectTomlMalformed: no error

What changed

  • pyproject.toml is read with go-toml's parser (already a dependency; the unstable AST gives each value's line). It reads [project].dependencies, each extra in [project.optional-dependencies], each PEP 735 [dependency-groups] group, and Poetry's dependencies, dev-dependencies and group.<name>.dependencies, including inline-table and dotted-key spellings. Scope is the table, for example project.optional-dependencies.dev. A malformed file is an error, which ParseFiles already reports per file.
  • LocalPurl.Dev (new, omitempty) says the manifest declares the dependency only for development: devDependencies in package.json and package-lock.json, Maven test scope, Gradle configurations starting with test or containing Test, and the Python rule below.
  • Python rule, in pkg/dependencies/internal/pyscope: every PEP 735 group, Poetry dev-dependencies and every Poetry group but main are dev (unchanged from the poetry.lock resolver). An extra is dev only when its PEP 685 name, or the part before its first dash, is a development name (dev, test(s), testing, lint, typing, mypy, doc(s), coverage, qa, ...), because users install extras such as socks at runtime.
  • The poetry.lock and uv.lock resolvers apply the same extra rule, so a dependency reads the same with or without a lockfile. Before, they treated every extra as runtime.
  • README gains a "Dev dependencies" section; CHANGELOG has Added and Fixed entries under Unreleased.

Testing

  • go test ./..., make fmt-check, go vet ./... and golangci-lint v2.14.0 run: all pass, 0 lint issues.
  • TestParsePyprojectToml covers the customer file, non-dependency strings, inline extras, PEP 735 with include-group, and Poetry tables.
  • TestDevDeclarations covers npm, package-lock, Maven and Gradle.
  • The poetry/dev-scope golden now declares ruff only through a lint extra; breaking the poetry rule fails that golden. uv/extras gains a test extra (iniconfig, dev) beside the runtime cli extra (click).

The pyproject.toml reader matched lines: any line holding "dependencies"
and "[" opened a section, so a one-line dependencies = ["requests>=2.31"]
was skipped, later strings such as readme = "README.md" became
dependencies, Poetry tables were misread and [dependency-groups] was never
read. No declaration said whether it was only for development.

Read the file with the go-toml parser and record the table each
dependency comes from as its Scope. Add LocalPurl.Dev, set from each
manifest's own dev marker: devDependencies, Maven's test scope, Gradle test
configurations, PEP 735 groups, Poetry dev groups, and extras named for
development (package internal/pyscope). The poetry.lock and uv.lock
resolvers use the same extra rule, so a dependency reads the same with or
without a lockfile.
@matiasdaloia
matiasdaloia merged commit 0fa44a6 into main Nov 6, 2026
9 checks passed
@matiasdaloia
matiasdaloia deleted the fix/pyproject-dependency-scope branch November 6, 2026 15:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant