Repository navigation
fix(dependencies): read pyproject.toml as TOML and mark dev declarations - #134
Merged
Merged
Conversation
The pyproject.toml reader matched lines: any line holding "dependencies" and "[" opened a section, so a one-line dependencies = ["requests>=2.31"] was skipped, later strings such as readme = "README.md" became dependencies, Poetry tables were misread and [dependency-groups] was never read. No declaration said whether it was only for development. Read the file with the go-toml parser and record the table each dependency comes from as its Scope. Add LocalPurl.Dev, set from each manifest's own dev marker: devDependencies, Maven's test scope, Gradle test configurations, PEP 735 groups, Poetry dev groups, and extras named for development (package internal/pyscope). The poetry.lock and uv.lock resolvers use the same extra rule, so a dependency reads the same with or without a lockfile.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A self-hosted Earnie customer scanned a Python project with no lockfile and got its dev tools (
pytest,pylint) back as ordinary runtime dependencies, so a policy that skips dev dependencies flaggedpylintas GPL. The manifest parser gave no way to tell a dev declaration from a runtime one, and itspyproject.tomlreader also lost or invented dependencies.What was broken
ParsePyprojectTomlmatched lines: any line holdingdependenciesand[opened a "section". On the customer's fileit returned
pytestandpylintwith no scope and droppedrequestsentirely. The new tests run against the old parser (v0.16.0) show the rest of the damage:What changed
pyproject.tomlis read with go-toml's parser (already a dependency; theunstableAST gives each value's line). It reads[project].dependencies, each extra in[project.optional-dependencies], each PEP 735[dependency-groups]group, and Poetry'sdependencies,dev-dependenciesandgroup.<name>.dependencies, including inline-table and dotted-key spellings.Scopeis the table, for exampleproject.optional-dependencies.dev. A malformed file is an error, whichParseFilesalready reports per file.LocalPurl.Dev(new,omitempty) says the manifest declares the dependency only for development:devDependenciesinpackage.jsonandpackage-lock.json, Maventestscope, Gradle configurations starting withtestor containingTest, and the Python rule below.pkg/dependencies/internal/pyscope: every PEP 735 group, Poetrydev-dependenciesand every Poetry group butmainare dev (unchanged from thepoetry.lockresolver). An extra is dev only when its PEP 685 name, or the part before its first dash, is a development name (dev,test(s),testing,lint,typing,mypy,doc(s),coverage,qa, ...), because users install extras such assocksat runtime.poetry.lockanduv.lockresolvers apply the same extra rule, so a dependency reads the same with or without a lockfile. Before, they treated every extra as runtime.Testing
go test ./...,make fmt-check,go vet ./...andgolangci-lint v2.14.0 run: all pass, 0 lint issues.TestParsePyprojectTomlcovers the customer file, non-dependency strings, inline extras, PEP 735 withinclude-group, and Poetry tables.TestDevDeclarationscovers npm, package-lock, Maven and Gradle.poetry/dev-scopegolden now declaresruffonly through alintextra; breaking the poetry rule fails that golden.uv/extrasgains atestextra (iniconfig, dev) beside the runtimecliextra (click).