-
-
Notifications
You must be signed in to change notification settings - Fork 98
feat: populate component.authors from pyproject.toml, Poetry manifests and packaging metadata #1096
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,83 @@ | ||
| # This file is part of CycloneDX Python | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
| # | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
| # Copyright (c) OWASP Foundation. All Rights Reserved. | ||
|
|
||
| """ | ||
| Helpers for turning the free-form "person" data found in `pyproject.toml`, | ||
| Poetry manifests and packaging core-metadata into `OrganizationalContact` model instances. | ||
| """ | ||
|
|
||
| from re import compile as re_compile | ||
| from typing import TYPE_CHECKING, Optional | ||
|
|
||
| from cyclonedx.model.contact import OrganizationalContact | ||
|
|
||
| if TYPE_CHECKING: # pragma: nocover | ||
| from collections.abc import Iterable | ||
|
|
||
| # Matches the "Name <email>" convention used by Poetry's `authors`/`maintainers` lists | ||
| # and by packaging core-metadata's free-text `Author`/`Author-email` fields. | ||
| # Both the name and the `<email>` part are optional on their own - see `person_string2contact()`. | ||
| _PERSON_STRING_MATCHER = re_compile(r'^\s*(?P<name>[^<]*?)\s*(?:<(?P<email>[^<>]*)>)?\s*$') | ||
|
|
||
|
|
||
| def person_string2contact(value: str) -> Optional[OrganizationalContact]: | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. why is this custom thing needed at all? from email.utils import parseaddr
name, addr = parseaddr(text) if '@' in text else (text,'') |
||
| """ | ||
| Parse a free-form ``"Name <email>"`` string - as used by Poetry and by | ||
| packaging core-metadata - into an `OrganizationalContact`. | ||
|
|
||
| The name and the email are each optional on their own: a bare name | ||
| (``"Jane Doe"``), a bare email (``"<jane@example.com>"``) and the | ||
| combined form (``"Jane Doe <jane@example.com>"``) are all valid. | ||
|
|
||
| Returns `None` if `value` carries no usable name or email at all. | ||
| """ | ||
| m = _PERSON_STRING_MATCHER.match(value) | ||
| if m is None: | ||
| # Reachable: the pattern requires the whole string to be a bare name, a bare | ||
| # `<email>`, or exactly one of each in that order - anything with more than | ||
| # one `<...>` fragment, an unbalanced bracket, or trailing text after a | ||
| # closing `>` does not match at all (see test_multiple_angle_bracket_fragments | ||
| # and friends). Real-world pyproject.toml/Poetry authors data occasionally has | ||
| # this shape; treat it the same as "no usable name or email" rather than crash. | ||
| return None | ||
| name = m.group('name') or None | ||
| email = m.group('email') or None | ||
|
|
||
| if name is None and email is None: | ||
| return None | ||
| return OrganizationalContact(name=name, email=email) | ||
|
|
||
|
|
||
| def contacts2author(contacts: 'Iterable[OrganizationalContact]') -> Optional[str]: | ||
| """ | ||
| Derive the legacy singular `Component.author` string from a set of `OrganizationalContact`. | ||
|
|
||
| CycloneDX 1.6 deprecated the singular free-text `author` in favour of the | ||
| structured, repeatable `authors`. There is no agreed-upon way to fold | ||
| multiple authors back into a single string - see | ||
| https://github.com/CycloneDX/specification/issues/335 - so this | ||
| deliberately does *not* guess a join convention for more than one author. | ||
|
|
||
| Returns the sole author's ``"Name <email>"``/``"Name"``/``"<email>"`` | ||
| representation when there is exactly one, else `None`. | ||
| """ | ||
| contacts = tuple(contacts) | ||
| if len(contacts) != 1: | ||
| return None | ||
| contact = contacts[0] | ||
| if contact.name and contact.email: | ||
| return f'{contact.name} <{contact.email}>' | ||
| return contact.name or contact.email or None | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -16,11 +16,13 @@ | |
| # Copyright (c) OWASP Foundation. All Rights Reserved. | ||
|
|
||
| from collections.abc import Generator | ||
| from email.utils import getaddresses | ||
| from re import compile as re_compile | ||
| from typing import TYPE_CHECKING | ||
|
|
||
| from cyclonedx.exception.model import InvalidUriException | ||
| from cyclonedx.model import AttachedText, ExternalReference, ExternalReferenceType, XsUri | ||
| from cyclonedx.model.contact import OrganizationalContact | ||
| from cyclonedx.model.license import DisjunctiveLicense, LicenseAcknowledgement | ||
|
|
||
| from .cdx import url_label_to_ert | ||
|
|
@@ -93,6 +95,45 @@ def metadata2extrefs(metadata: 'PackageMetadata') -> Generator['ExternalReferenc | |
| pass | ||
|
|
||
|
|
||
| def metadata2authors(metadata: 'PackageMetadata') -> Generator['OrganizationalContact', None, None]: | ||
| """ | ||
| See: | ||
| - https://packaging.python.org/en/latest/specifications/core-metadata/#author | ||
| - https://packaging.python.org/en/latest/specifications/core-metadata/#author-email | ||
|
|
||
| `Author` and `Author-email` are two independent free-text fields; there is no | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
| guaranteed way to correlate them when either holds more than one person. So: | ||
| - if `Author-email` resolves to exactly one address with no display name of its | ||
| own, and `Author` looks like a bare name (no `<`/`@`), the two are combined | ||
| into a single contact; | ||
| - otherwise, every address found in `Author-email` becomes its own contact, | ||
| and a bare-name `Author` is used as a fallback contact only when | ||
| `Author-email` is absent entirely. | ||
| """ | ||
| author = metadata.get('Author') | ||
| author_email = metadata.get('Author-email') | ||
| if author_email: | ||
| # `email.utils.getaddresses()` expects RFC 5322 address syntax. Fed something | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. please remove those comments - they only describe how |
||
| # that isn't actually shaped like an email - e.g. a bare name with no `<...>` | ||
| # and no `@` - it silently mis-splits on whitespace and drops everything but | ||
| # the last "word": `getaddresses(['Jane Doe']) == [('', 'Jane')]`. | ||
| # Guard against that by only trusting entries that actually look like an email. | ||
| addresses = [ | ||
| (name or None, email) | ||
| for name, email in getaddresses([author_email]) | ||
| if '@' in email | ||
| ] | ||
| bare_name = bool(author and '<' not in author and '@' not in author) | ||
| if len(addresses) == 1 and addresses[0][0] is None and bare_name: | ||
| yield OrganizationalContact(name=author, email=addresses[0][1]) | ||
| return | ||
| for name, email in addresses: | ||
| yield OrganizationalContact(name=name, email=email) | ||
| return | ||
|
Comment on lines
+130
to
+132
|
||
| if author: | ||
| yield OrganizationalContact(name=author) | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. how comes? python packaging's 'author` clearly corresponds to CycloneDX's author - bot are arbitrary strings ... |
||
|
|
||
|
|
||
| _NORMALIZE_PN_MATCHER = re_compile(r'[-_.]+') | ||
| _NORMALIZE_PN_REPLACE = '-' | ||
|
|
||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -32,10 +32,12 @@ | |
| from cyclonedx.exception.model import InvalidUriException | ||
| from cyclonedx.model import AttachedText, Encoding, ExternalReference, XsUri | ||
| from cyclonedx.model.component import Component | ||
| from cyclonedx.model.contact import OrganizationalContact | ||
| from cyclonedx.model.license import DisjunctiveLicense, LicenseAcknowledgement | ||
| from packaging.requirements import Requirement | ||
|
|
||
| from .cdx import url_label_to_ert | ||
| from .contact import contacts2author, person_string2contact | ||
| from .license_trove_classifier import is_license_trove, license_trove2spdx | ||
| from .mimetypes import guess_type | ||
|
|
||
|
|
@@ -114,15 +116,41 @@ def project2extrefs(project: dict[str, Any]) -> Generator['ExternalReference', N | |
| pass | ||
|
|
||
|
|
||
| def project2authors(project: dict[str, Any]) -> Generator['OrganizationalContact', None, None]: | ||
| # see https://packaging.python.org/en/latest/specifications/pyproject-toml/#authors-maintainers | ||
| # see https://peps.python.org/pep-0621/#authors-maintainers | ||
| for author in project.get('authors', ()): | ||
| if isinstance(author, str): | ||
| # Not per spec -- PEP 621 authors are tables, not strings -- but some | ||
| # real-world pyproject.toml files use Poetry's "Name <email>" convention | ||
| # here regardless. Be lenient and parse it the same way, rather than crash. | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. nope. we will not be lenient here! File is called "pip621" and docs say also...
this thing shall implement the PEP621 only, not some "in the wild" observations. |
||
| contact = person_string2contact(author) | ||
| if contact is not None: | ||
| yield contact | ||
| continue | ||
| if not isinstance(author, dict): | ||
| # Not per spec at all -- e.g. `authors = [123]` -- TOML happily allows it, | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. nope. we will not be lenient here! File is called "pip621" and docs say also...
this thing shall implement the PEP621 only, not some "in the wild" observations. |
||
| # PEP 621 does not. There is nothing name/email-shaped to extract; skip it | ||
| # rather than crash on `author.get(...)`. | ||
| continue | ||
| name = author.get('name') or None | ||
| email = author.get('email') or None | ||
| if name is not None or email is not None: | ||
| yield OrganizationalContact(name=name, email=email) | ||
|
|
||
|
|
||
| def project2component(project: dict[str, Any], *, | ||
| ctype: 'ComponentType') -> 'Component': | ||
| dynamic = project.get('dynamic', ()) | ||
| authors = tuple(project2authors(project)) if 'authors' not in dynamic else () | ||
| return Component( | ||
| type=ctype, | ||
| name=project['name'], | ||
| version=project.get('version', None) if 'version' not in dynamic else None, | ||
| description=project.get('description', None) if 'description' not in dynamic else None, | ||
| external_references=project2extrefs(project), | ||
| authors=authors, | ||
| author=contacts2author(authors), | ||
| # licenses are not gathered here per default, they may be sourced otherwise | ||
| # TODO add more properties according to spec | ||
| ) | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -29,10 +29,12 @@ | |
| from cyclonedx.factory.license import LicenseFactory | ||
| from cyclonedx.model import ExternalReference, ExternalReferenceType, XsUri | ||
| from cyclonedx.model.component import Component | ||
| from cyclonedx.model.contact import OrganizationalContact | ||
| from cyclonedx.model.license import LicenseAcknowledgement | ||
| from packaging.requirements import Requirement | ||
|
|
||
| from .cdx import licenses_fixup, url_label_to_ert | ||
| from .contact import contacts2author, person_string2contact | ||
| from .pep621 import classifiers2licenses | ||
|
|
||
| if TYPE_CHECKING: | ||
|
|
@@ -62,13 +64,29 @@ def poetry2extrefs(poetry: dict[str, Any]) -> Generator['ExternalReference', Non | |
| pass | ||
|
|
||
|
|
||
| def poetry2authors(poetry: dict[str, Any]) -> Generator['OrganizationalContact', None, None]: | ||
| # see https://python-poetry.org/docs/pyproject/#authors | ||
| for author in poetry.get('authors', ()): | ||
| if not isinstance(author, str): | ||
| # Not per spec -- Poetry's `authors` is a list of "Name <email>" strings -- | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
which spec? anyway,just remove this comment. it has no value - it literally describes the code |
||
| # but e.g. `authors = [123]` is valid TOML. `person_string2contact()` | ||
| # requires a string; skip anything else rather than crash. | ||
| continue | ||
| contact = person_string2contact(author) | ||
| if contact is not None: | ||
| yield contact | ||
|
|
||
|
|
||
| def poetry2component(poetry: dict[str, Any], *, ctype: 'ComponentType') -> 'Component': | ||
| authors = tuple(poetry2authors(poetry)) | ||
| component = Component( | ||
| type=ctype, | ||
| name=poetry['name'], | ||
| version=poetry.get('version'), | ||
| description=poetry.get('description'), | ||
| external_references=poetry2extrefs(poetry), | ||
| authors=authors, | ||
| author=contacts2author(authors), | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. arguable - as described in the ticket for this PR. I'd rather go with |
||
| # TODO add more properties according to spec | ||
| ) | ||
| # region licenses | ||
|
|
||
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
arguable - as described in the ticket for this PR.
see #648 (comment)
I'd rather go with
' & '.join(...))