HADOOP-19974. Publish CycloneDX 1.7 SBOMs that mark shaded dependencies

[ppkarwasz]

Description of PR

Every JAR module now attaches a CycloneDX 1.7 XML SBOM under the dist profile (the profile used to deploy artifacts to Maven Central, see HADOOP-18590).

CycloneDX 1.7 introduces an isExternal attribute that makes it possible to distinguish dependencies shaded inside a JAR from those merely referenced on the classpath — a distinction essential for vulnerability management. Since cyclonedx-maven-plugin does not yet support maven-shade-plugin (see CycloneDX/cyclonedx-maven-plugin#472), this PR post-processes the generated BOM with a Groovy script (dev-support/sbom/sbom.groovy, run by gmavenplus-plugin in the verify phase):

• Ordinary dependencies get an explicit isExternal="true"; dependencies embedded by maven-shade-plugin rely on the schema default of false.
• Relocated dependencies are technically derivative libraries, so they receive a pedigree ancestor carrying the original artifact's purl and SHA-256 hash. Both the relocated copy and its ancestor deliberately share the original's PURL, because vulnerability scanners cannot match a newly minted PURL against vulnerability databases and do not yet consult the pedigree.
• Dependencies embedded verbatim (modules shading without relocations) keep the original's hash and get a notes-only pedigree documenting the embedding.
• The shaded artifact set is derived from the effective maven-shade-plugin configuration, so it cannot drift from the shade artifactSet; -DskipShade builds classify everything as external, matching the jar actually produced.
• Hash lists are trimmed to SHA-256 and the result is re-validated against the CycloneDX 1.7 schema, failing the build on violations.

Out of scope (follow-ups): SBOMs for the binary distribution tarball, correcting license metadata, and a possible switch to JSON output (or publishing both formats).

How was this patch tested?

mvn clean install -Pdist -DskipTests on representative modules, asserting the generated target/bom.xml:

• hadoop-annotations (no shading): all components isExternal="true", no pedigree.
• hadoop-bos (include-style artifactSet with relocations): only the 3 compile-scope includes embedded, with pedigree ancestors carrying the SHA-256; provided-scope includes correctly reported external — verified against the actual jar contents.
• hadoop-tos (no artifactSet, no relocations): all 5 compile-scope dependencies embedded verbatim, hashes kept, notes-only pedigree.
• hadoop-client-runtime / hadoop-client-api (exclude-style with wildcards / include-style): all 63 externals match the shade excludes (incl. io.netty:* and org.glassfish.* wildcard families); 70 embedded components carry relocation pedigrees.
• -DskipShade and -Dcyclonedx.skip paths verified as no-ops.
• Schema validity is enforced by the script itself on every module.

For code changes:

• Does the title of this PR start with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
• Object storage: Have the integration tests been executed and the endpoint
  declared according to the connector-specific documentation? Note: Automated CI
  testing doesn't cover all cases so manual testing with cloud storage is still
  required.
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0? (build-time only: gmavenplus-plugin, org.apache.groovy:groovy and org.cyclonedx:cyclonedx-core-java are all Apache-2.0; nothing is bundled in released artifacts, so LICENSE-binary/NOTICE-binary are unaffected)
• If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

AI Tooling

If an AI tool was used:

• The PR includes the phrase "Contains content generated by Claude Code"
• My use of AI contributions follows the ASF legal policy
  https://www.apache.org/legal/generative-tooling.html

[ppkarwasz]

For reviewers, here are three real entries taken from the generated BOMs (<description> and <externalReferences> elided for brevity).

1. External dependency — commons-logging in hadoop-client-runtime/target/bom.xml. It is on the shade artifactSet exclude list (left unshaded so downstream users can configure logging), so it gets an explicit isExternal="true" and keeps its hash:

<component type="library" bom-ref="pkg:maven/commons-logging/commons-logging@1.3.0?type=jar" isExternal="true">
  <publisher>The Apache Software Foundation</publisher>
  <group>commons-logging</group>
  <name>commons-logging</name>
  <version>1.3.0</version>
  <scope>required</scope>
  <hashes>
    <hash alg="SHA-256">66d3c980470b99b0c511dad3dfc0ae7b265ec1fb144e96bc0253a8a175fd34d9</hash>
  </hashes>
  <licenses>
    <license>
      <id>Apache-2.0</id>
      <url>https://www.apache.org/licenses/LICENSE-2.0</url>
    </license>
  </licenses>
  <purl>pkg:maven/commons-logging/commons-logging@1.3.0?type=jar</purl>
</component>

2. Non-relocated embedded dependency — httpclient5 in hadoop-tos/target/bom.xml. hadoop-tos shades without relocations, so the embedded classes are verbatim copies: the component omits isExternal (the schema default is false), keeps the original's hash, and carries a notes-only pedigree:

<component type="library" bom-ref="pkg:maven/org.apache.httpcomponents.client5/httpclient5@5.5?type=jar">
  <publisher>The Apache Software Foundation</publisher>
  <group>org.apache.httpcomponents.client5</group>
  <name>httpclient5</name>
  <version>5.5</version>
  <scope>required</scope>
  <hashes>
    <hash alg="SHA-256">496b4b0e8d5f3a8139a5d2638486d304758bac3a9c39d76989f663cfd9354fc9</hash>
  </hashes>
  <licenses>
    <license>
      <id>Apache-2.0</id>
    </license>
  </licenses>
  <purl>pkg:maven/org.apache.httpcomponents.client5/httpclient5@5.5?type=jar</purl>
  <pedigree>
    <notes>Embedded verbatim into this artifact by maven-shade-plugin, without package relocation; the hashes describe the original standalone artifact.</notes>
  </pedigree>
</component>

3. Relocated embedded dependency — commons-lang3 in hadoop-client-runtime/target/bom.xml. The embedded copy is a rewritten derivative (relocated to org.apache.hadoop.shaded), so the component carries no hash of its own; the SHA-256 of the original standalone artifact sits on the pedigree ancestor, which deliberately shares the component's PURL so vulnerability scanners can still match it:

<component type="library" bom-ref="pkg:maven/org.apache.commons/commons-lang3@3.20.0?type=jar">
  <publisher>The Apache Software Foundation</publisher>
  <group>org.apache.commons</group>
  <name>commons-lang3</name>
  <version>3.20.0</version>
  <scope>optional</scope>
  <licenses>
    <license>
      <id>Apache-2.0</id>
      <url>https://www.apache.org/licenses/LICENSE-2.0</url>
    </license>
  </licenses>
  <purl>pkg:maven/org.apache.commons/commons-lang3@3.20.0?type=jar</purl>
  <pedigree>
    <ancestors>
      <component type="library">
        <group>org.apache.commons</group>
        <name>commons-lang3</name>
        <version>3.20.0</version>
        <hashes>
          <hash alg="SHA-256">69e5c9fa35da7a51a5fd2099dfe56a2d8d32cf233e2f6d770e796146440263f4</hash>
        </hashes>
        <purl>pkg:maven/org.apache.commons/commons-lang3@3.20.0?type=jar</purl>
      </component>
    </ancestors>
    <notes>Relocated and embedded into this artifact by maven-shade-plugin; the ancestor component describes the original standalone artifact.</notes>
  </pedigree>
</component>

[steveloughran]

Looks useful. Let's see what yetus says.