Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,25 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Added

- **`LocalPurl.Dev`** says the manifest declares a dependency only to develop the project:
`devDependencies` in `package.json` and `package-lock.json`, Maven's `test` scope, Gradle test
configurations, and the `pyproject.toml` groups and development extras. `dependencies` output
gains `"dev": true` on those entries.

### Fixed

- **`pyproject.toml` is read as TOML.** The line-based reader dropped a one-line
`dependencies = ["requests>=2.31"]`, invented dependencies from other strings such as
`readme = "README.md"`, misread Poetry tables, and never read `[dependency-groups]`. It now
reads `[project]` dependencies, each extra in `[project.optional-dependencies]`, each PEP 735
group and Poetry's dependencies, dev-dependencies and groups, with the table as `Scope`, such as
`project.optional-dependencies.dev`. A malformed file is now an error instead of a partial list.
- **Development extras are dev in `poetry.lock` and `uv.lock` graphs.** An extra named for
development (`dev`, `test`, `lint`, `docs` and the like; see `pkg/dependencies/README.md`) marks
its dependencies dev, as the manifest parser does. Other extras stay runtime.

## [0.16.0] - 2026-10-30

### Added
Expand Down
31 changes: 28 additions & 3 deletions pkg/dependencies/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,7 @@ type LocalPurl struct {
Purl string // Package URL (e.g., "pkg:npm/react@18.0.0")
Requirement string // Version requirement (e.g., ">=1.0.0")
Scope string // Dependency scope (e.g., "devDependencies")
Dev bool // The manifest declares it only to develop the project
}
```

Expand Down Expand Up @@ -208,7 +209,10 @@ pkg:nuget/Newtonsoft.Json@13.0.1
- Filters out URLs, paths, and recursive dependencies
- Handles operators: `==`, `>=`, `~`, `!=`, etc.
- Exact versions (`==`) included in PURL, others in `Requirement`
- **pyproject.toml**: Parses dependencies from `[tool.poetry.dependencies]` section
- **pyproject.toml**: Read as TOML: `[project]` dependencies, each extra in
`[project.optional-dependencies]`, each PEP 735 `[dependency-groups]` group, and Poetry's
`dependencies`, `dev-dependencies` and groups. `Scope` is the table, such as
`project.optional-dependencies.dev`. See [Dev dependencies](#dev-dependencies) for which are dev.

### Maven Parser

Expand Down Expand Up @@ -246,6 +250,27 @@ pkg:nuget/Newtonsoft.Json@13.0.1
- **packages.config**: Parses legacy package elements
- Both extract package ID and version

### Dev dependencies

`LocalPurl.Dev` is true when the manifest declares a dependency only to develop the project
(tests, linting, type checking, documentation, build tooling), so a consumer can tell it apart
without a lockfile:

| Manifest | Dev |
|---|---|
| `package.json` | `devDependencies` |
| `package-lock.json` | entries marked `"dev": true` |
| `pom.xml` | `test` scope (`provided` is runtime: the platform supplies it) |
| `build.gradle` | configurations that start with `test` or contain `Test`, such as `testImplementation` and `androidTestImplementation` |
| `pyproject.toml` | every `[dependency-groups]` group, Poetry's `dev-dependencies` and every Poetry group but `main`, and every development extra |

An extra in `[project.optional-dependencies]` is installed by any user who asks for it, so it is
dev only when its name says so: its PEP 685 normalized name, or the part before its first dash,
is one of `dev`, `devel`, `develop`, `development`, `test`, `tests`, `testing`, `lint`,
`linting`, `typing`, `mypy`, `doc`, `docs`, `coverage` or `qa`. `dev`, `Test_Integration` and
`docs-build` are dev; `socks` and `devices` are not. The `poetry.lock` and `uv.lock` resolvers
apply the same rule, so a dependency reads the same with or without a lockfile.

## Dependency Graphs

The parsers above return a flat list of PURLs per file. To know which package
Expand Down Expand Up @@ -298,8 +323,8 @@ and edges kept, and a bad lockfile is reported on its root, never as a panic.
| `pnpm-lock.yaml` (5.x, 6.x, 9.x) | lockfile only | every importer | each importer's `devDependencies` |
| `Cargo.lock` | lockfile, `Cargo.toml` and workspace member manifests | `Cargo.toml` and each workspace member; a path crate that something depends on is a local node | `[dev-dependencies]`, including per target, that are not also normal or build dependencies; unknown for a root without its manifest |
| `Gemfile.lock` | lockfile and `Gemfile` | `Gemfile`; `PATH` gems are local nodes | gems declared only in the `development` and `test` groups; unknown without a `Gemfile` |
| `poetry.lock` | lockfile and `pyproject.toml` | project; path dependencies are local nodes | `dev-dependencies`, every Poetry group but `main`, and `[dependency-groups]`; unknown without `pyproject.toml` |
| `uv.lock` (version 1) | lockfile only | the project and each workspace member, at its `pyproject.toml`; editable, directory and path packages are local nodes | each root's `dev-dependencies` groups; extras are runtime |
| `poetry.lock` | lockfile and `pyproject.toml` | project; path dependencies are local nodes | `dev-dependencies`, every Poetry group but `main`, every `[dependency-groups]` group, and development extras; unknown without `pyproject.toml` |
| `uv.lock` (version 1) | lockfile only | the project and each workspace member, at its `pyproject.toml`; editable, directory and path packages are local nodes | each root's `dev-dependencies` groups and development extras |
| `packages.lock.json` (NuGet v1, v2) | lockfile, and the name of the project file beside it | the `.csproj`, `.fsproj` or `.vbproj` beside it, or the lockfile when there is not exactly one; every target framework is merged, and `Project` entries are local nodes | unknown: the lockfile does not record them |
| `composer.lock` | lockfile and `composer.json` | project; path repository packages are local nodes | `require-dev`; unknown without `composer.json` |
| `gradle.lockfile` | lockfile, and whether `build.gradle.kts` is beside it | `build.gradle`, or `build.gradle.kts` when it is the only build file; no direct members and no edges (`edges_unknown`) | unknown |
Expand Down
63 changes: 63 additions & 0 deletions pkg/dependencies/internal/pyscope/pyscope.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
// SPDX-License-Identifier: MIT
/*
* Copyright (c) 2026, SCANOSS
*
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the "Software"), to deal
* in the Software without restriction, including without limitation the rights
* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is
* furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*/

// Package pyscope decides which pyproject.toml dependencies are only needed to
// develop the project. The manifest parser and the poetry.lock and uv.lock
// resolvers share it, so a dependency reads as dev with or without a lockfile.
//
// Every PEP 735 [dependency-groups] group, Poetry's dev-dependencies and every
// Poetry group but main are for development: none of them is installed with
// the package. An extra is installed by any user who asks for it, so it is
// dev only when its name says it is (DevExtra).
package pyscope

import (
"regexp"
"strings"
)

// devExtraNames are the extra names the Python ecosystem uses for tests,
// linting, type checking, documentation and other development tooling.
var devExtraNames = map[string]bool{
"dev": true, "devel": true, "develop": true, "development": true,
"test": true, "tests": true, "testing": true,
"lint": true, "linting": true, "typing": true, "mypy": true,
"doc": true, "docs": true, "coverage": true, "qa": true,
}

var separators = regexp.MustCompile(`[-_.]+`)

// DevExtra reports whether an extra is for development: its normalized name
// (PEP 685), or the part of it before the first dash, is a development name,
// so dev, Test, docs and test-integration are dev, and socks and devices are
// not.
func DevExtra(name string) bool {
n := separators.ReplaceAllString(strings.ToLower(strings.TrimSpace(name)), "-")
head, _, _ := strings.Cut(n, "-")
return devExtraNames[n] || devExtraNames[head]
}

// DevPoetryGroup reports whether a Poetry dependency group is for development.
func DevPoetryGroup(name string) bool {
return name != "main"
}
49 changes: 49 additions & 0 deletions pkg/dependencies/internal/pyscope/pyscope_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
// SPDX-License-Identifier: MIT
/*
* Copyright (c) 2026, SCANOSS
*
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the "Software"), to deal
* in the Software without restriction, including without limitation the rights
* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is
* furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*/

package pyscope

import "testing"

func TestDevExtra(t *testing.T) {
t.Parallel()
for name, want := range map[string]bool{
"dev": true, "Dev": true, "tests": true, "test-integration": true, "Test_Integration": true,
"docs.build": true, "lint": true, "typing": true,
"socks": false, "devices": false, "testbed": false, "all": false, "postgres": false, "": false,
} {
if got := DevExtra(name); got != want {
t.Errorf("DevExtra(%q) = %v, want %v", name, got, want)
}
}
}

func TestDevPoetryGroup(t *testing.T) {
t.Parallel()
if DevPoetryGroup("main") {
t.Error("main is the runtime group")
}
if !DevPoetryGroup("test") {
t.Error("every group but main is for development")
}
}
110 changes: 110 additions & 0 deletions pkg/dependencies/parsers/dev_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
// SPDX-License-Identifier: MIT
/*
* Copyright (c) 2026, SCANOSS
*
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the "Software"), to deal
* in the Software without restriction, including without limitation the rights
* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is
* furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*/

package parsers

import (
"maps"
"testing"
)

// TestDevDeclarations checks that each manifest's own way of declaring a
// development-only dependency sets Dev, and that a runtime one does not.
func TestDevDeclarations(t *testing.T) {
t.Parallel()

tests := []struct {
name string
parse ParserFunc
file string
content string
want map[string]bool // purl -> Dev
}{
{
name: "package.json devDependencies",
parse: ParsePackageJSON,
file: "package.json",
content: `{"dependencies": {"express": "^4.19.0"}, "devDependencies": {"jest": "^29.7.0"}}`,
want: map[string]bool{"pkg:npm/express@^4.19.0": false, "pkg:npm/jest@^29.7.0": true},
},
{
name: "package-lock.json dev entries",
parse: ParsePackageLock,
file: "package-lock.json",
content: `{"lockfileVersion": 3, "packages": {
"": {"name": "app"},
"node_modules/express": {"version": "4.19.2"},
"node_modules/jest": {"version": "29.7.0", "dev": true}
}}`,
want: map[string]bool{"pkg:npm/express@4.19.2": false, "pkg:npm/jest@29.7.0": true},
},
{
name: "pom.xml test scope",
parse: ParsePomXML,
file: "pom.xml",
content: `<project><dependencies>
<dependency><groupId>org.slf4j</groupId><artifactId>slf4j-api</artifactId><version>2.0.9</version></dependency>
<dependency><groupId>junit</groupId><artifactId>junit</artifactId><version>4.13.2</version><scope>test</scope></dependency>
<dependency><groupId>javax.servlet</groupId><artifactId>servlet-api</artifactId><version>2.5</version><scope>provided</scope></dependency>
</dependencies></project>`,
want: map[string]bool{
"pkg:maven/org.slf4j/slf4j-api@2.0.9": false,
"pkg:maven/junit/junit@4.13.2": true,
"pkg:maven/javax.servlet/servlet-api@2.5": false,
},
},
{
name: "build.gradle test configurations",
parse: ParseBuildGradle,
file: "build.gradle",
content: `dependencies {
implementation 'com.google.guava:guava:32.1.2-jre'
testImplementation 'junit:junit:4.13.2'
androidTestImplementation group: 'org.mockito', name: 'mockito-core', version: '5.5.0'
}
`,
want: map[string]bool{
"pkg:maven/com.google.guava/guava@32.1.2-jre": false,
"pkg:maven/junit/junit@4.13.2": true,
"pkg:maven/org.mockito/mockito-core@5.5.0": true,
},
},
}

for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
result, err := tc.parse([]byte(tc.content), tc.file)
if err != nil {
t.Fatalf("parse: %v", err)
}
got := map[string]bool{}
for _, p := range result.Purls {
got[p.Purl] = p.Dev
}
if !maps.Equal(got, tc.want) {
t.Errorf("got purl -> Dev %v, want %v", got, tc.want)
}
})
}
}
8 changes: 8 additions & 0 deletions pkg/dependencies/parsers/gradle.go
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,7 @@ func ParseBuildGradle(fileContent []byte, filePath string) (*LocalDependency, er
Purl: purl,
Requirement: version,
Scope: scope,
Dev: gradleTestConfiguration(scope),
Line: lineNo,
DeclaredText: strings.TrimSpace(originalLine),
})
Expand Down Expand Up @@ -169,6 +170,7 @@ func ParseBuildGradle(fileContent []byte, filePath string) (*LocalDependency, er
Purl: purl,
Requirement: version,
Scope: currentScope,
Dev: gradleTestConfiguration(currentScope),
Line: depLine,
DeclaredText: strings.TrimSpace(line),
})
Expand Down Expand Up @@ -197,3 +199,9 @@ func ParseGradle(fileContent []byte, filePath string) (*LocalDependency, error)
Purls: []LocalPurl{},
}, nil
}

// gradleTestConfiguration reports whether a dependency configuration only
// builds or runs tests, such as testImplementation or androidTestImplementation.
func gradleTestConfiguration(configuration string) bool {
return strings.HasPrefix(configuration, "test") || strings.Contains(configuration, "Test")
}
1 change: 1 addition & 0 deletions pkg/dependencies/parsers/maven.go
Original file line number Diff line number Diff line change
Expand Up @@ -291,6 +291,7 @@ func ParsePomXML(fileContent []byte, filePath string) (*LocalDependency, error)
Purl: purl,
Requirement: version,
Scope: scope,
Dev: scope == "test",
Line: offsetToLine(fileContent, cd.startOff),
DeclaredText: strings.TrimSpace(string(fileContent[cd.startOff:cd.endOff])),
})
Expand Down
2 changes: 2 additions & 0 deletions pkg/dependencies/parsers/npm.go
Original file line number Diff line number Diff line change
Expand Up @@ -224,6 +224,7 @@ func ParsePackageJSON(fileContent []byte, filePath string) (*LocalDependency, er
Purl: purl,
Requirement: e.value,
Scope: scope,
Dev: scope == "devDependencies",
Line: offsetToLine(fileContent, e.off),
DeclaredText: lineTextAt(fileContent, e.off),
})
Expand Down Expand Up @@ -351,6 +352,7 @@ func parsePackageLockV2TokenWalk(fileContent []byte, result *LocalDependency) (*
Purl: purl,
Requirement: pkg.Version,
Scope: scope,
Dev: pkg.Dev,
Line: line,
DeclaredText: declText,
})
Expand Down
Loading
Loading