[FScholPer]

S-CORE Test Compatibility Kit (TCK) — Design & Delivery Plan

This plan describes how to build an s-core TCK — a portable, downloadable
test kit that lets a vendor (or integrator) verify whether their distribution
of the SCORE automotive middleware is compatible with the main platform
requirements: public APIs, SBOM / license policy, and behavioral
conformance. It adapts the Jakarta EE Platform TCK concept
(jakarta-jakartaeetck-11.0.1.zip, Jakarta-Platform-TCK-Users-Guide.md) to
the SCORE reference-integration stack (Bazel + known_good.json + SPDX SBOM +
platform integration tests).

---

1. Goals & non-goals

Goals

• Provide a single downloadable binary (score-tck) produced by the
  reference-integration CI, attachable to a GitHub Release on tag (same flow as
  today's test-report zip in test_and_docs.yml).
• Let a vendor point that binary at their local distribution and get a
  machine-readable + human-readable compatibility report with a clear
  pass/fail verdict per check.
• Reuse existing infrastructure wherever possible: the SPDX SBOM target, the
  requirement-mapped platform integration tests, and the showcase CLI.
• Define "compatible" as an explicit, versioned set of conformance criteria
  (the s-core analogue of a Jakarta EE specification).

Non-goals (v1)

• Not a formal certification body / legal sign-off process (that is a policy
  layer on top; the TCK only produces evidence).
• Not a re-implementation of unit tests. We drive existing test assets.
• Not multi-OS in v1: target linux-x86_64 first (matches current CI), then
  extend to aarch64, QNX, eBCLFSA.

---

2. Conceptual mapping: Jakarta EE TCK → s-core TCK

Jakarta EE Platform TCK	s-core TCK equivalent	Notes / where it lives today
Specification (Jakarta EE 11 Spec)	s-core platform requirements: public API surface + SBOM/license policy + requirement IDs (stkh_req__*)	Requirement IDs already used in platform_integration_tests/** via @add_test_properties(partially_verifies=[...]).
VI (Vendor Implementation under test)	A vendor's distribution of s-core (their Bazel workspace, prebuilt libs+headers, or image)	The thing the TCK is run against.
CI (Compatible / reference implementation)	The reference integration build pinned by known_good.json — our "golden" distribution	Built in CI; source of golden SBOM + API baselines.
Porting Package (adapter the VI implements to plug into the harness)	A distribution manifest + adapter hooks (tck.config) that tell the runner how to locate artifacts, emit an SBOM, and build/run showcases for this distribution	New; the key abstraction (see §5).
ts.jte environment file	tck.config.toml / .json describing the distribution under test	New.
Test harness (JavaTest / JUnit5 / Arquillian)	The score-tck runner binary that orchestrates checks and writes a report	New; Python zipapp or static Rust binary.
TCK zip (bin/, artifacts/, guides/, src/tcks/apis/<api>/)	score-tck-<version>.tar.gz: bin/score-tck, reference/ (golden SBOM, API baselines), checks/, docs/	New; produced & attached in CI.
Test reports / certification	Compatibility report (JSON + HTML) with per-check pass/fail and an overall verdict	New output format (§8).

The single most important idea carried over from Jakarta is the porting
package: the TCK tests are written once, against a small adapter contract;
each distribution supplies its own implementation of that contract. For s-core
that contract is deliberately thin — a manifest plus a few hooks — because our
"tests" are largely existing assets (SBOM target, platform integration
tests) rather than a bespoke Java test suite.

---

3. What "compatible with the main platform requirements" means

We define conformance as three independent check families. A distribution is
conformant when all required checks in its declared profile pass.

3.1 SBOM / supply-chain conformance (explicitly requested)

The distribution must be able to produce an SPDX SBOM, and that SBOM must
satisfy the platform policy:

• Presence & format: valid SPDX (JSON), correct creationInfo, a root
  package matching the declared component name.
• Required components present: every module in known_good.json →
  sbom.tracked_modules (score_baselibs, score_communication,
  score_persistency, score_kyron, score_lifecycle, score_logging,
  score_time) appears as a component.
• No forbidden / unexpected components: diff against the golden SBOM; flag
  additions not in an allow-list (catches dropped-in third-party deps).
• License policy: every component's license is in the allowed set; no
  copyleft/forbidden licenses per policy file.
• Version/hash drift (informational → required over time): report components
  whose version or checksum differs from the golden baseline.

The reference build already emits this: bazel build //:sbom →
bazel-bin/sbom.spdx.json (component score_reference_integration). That is
our golden SBOM.

3.2 API conformance

The distribution's public API surface must match the platform baseline. Two
complementary mechanisms, from cheap to strong:

• Static API baseline diff (fast, no toolchain needed): compare exported
  symbols / header signatures against a generated baseline captured from the
  reference build. For C++: a normalized set of public declarations (e.g. via a
  libclang/c-index dump or nm-based export list). For Rust: the public
  crate API (cargo metadata + re-exported items) and/or exported symbols.
  Missing/renamed public symbols = fail; added symbols = warning.
• Behavioral API conformance (strong, needs toolchain): run the existing
  platform integration tests against the distribution (§3.3). These already
  exercise real APIs per requirement ID and are the authoritative signal that
  "the API works as the platform requires."

3.3 Behavioral / functional conformance (reuse existing PITs)

platform_integration_tests/ already encodes, per module, a showcase-driven
test mapped to a requirement:

• logging/test_logging_showcase.py → stkh_req__dev_experience__logging_support
• communication/, lifecycle/, time/, kyron/ → their respective req IDs

Each runs the Rust showcase CLI (//showcases/bin/cli) and asserts success.
The TCK re-runs exactly these against the vendor's distribution. Passing them is
the strongest evidence of API + behavioral compatibility, because they are the
same tests that gate the reference platform.

Optional (phase 2): also drive feature_integration_tests/ (ITF / C++ & Rust
test scenarios) for deeper functional coverage.

---

4. TCK package layout (the downloadable artifact)

score-tck-<version>.tar.gz
└── score-tck-<version>/
    ├── bin/
    │   └── score-tck                 # self-contained runner (zipapp or static binary)
    ├── reference/                    # the "CI" golden data, captured from a known_good build
    │   ├── sbom.spdx.json            # golden SBOM (from //:sbom)
    │   ├── api-baseline/             # per-module exported-symbol / header baselines
    │   │   ├── <module>.symbols.txt
    │   │   └── ...
    │   ├── requirements.json         # requirement-ID → {check, severity} mapping
    │   └── policy/
    │       ├── allowed-licenses.json
    │       └── forbidden-components.json
    ├── checks/                       # check implementations (bundled with the runner)
    │   ├── sbom_check.py
    │   ├── api_check.py
    │   └── behavioral_check.py
    ├── docs/
    │   ├── USER_GUIDE.md             # "how to run the TCK against your distribution"
    │   └── CONFORMANCE_CRITERIA.md   # the versioned definition of "compatible"
    └── examples/
        └── tck.config.example.toml

The runner is self-contained (no network, no Bazel required for the SBOM /
API checks) so a vendor can run it on an air-gapped machine. Only the
behavioral check needs a working toolchain + the distribution's build.

---

5. The porting package: tck.config (distribution manifest)

This is the s-core analogue of Jakarta's porting package. A vendor describes
their distribution once; the runner adapts to it.

# tck.config.toml — describes the distribution under test (the "VI")
[dist]
name = "acme-score-dist"
version = "1.4.2"
arch = "x86_64-linux"          # x86_64-linux | aarch64-linux | qnx-aarch64 | ...
profile = "full"               # full | web-profile-analogue (subset of checks)

[dist.paths]
# Where the prebuilt artifacts live (for SBOM + API static checks).
headers_dir = "./out/include"
libs_dir    = "./out/lib"
sbom_file   = "./out/sbom.spdx.json"     # vendor-generated SPDX, if they have one

[dist.build]                   # only needed for the behavioral check
# How to build & run showcases against THIS distribution.
build_cmd  = "bazel build --config=x86_64 //showcases/bin/cli"
run_cli    = "./bazel-bin/showcases/bin/cli"
examples   = ["LoggingApp Demo", "ComApiExample"]   # subset of showcase examples

[checks]
sbom        = true
api_static  = true
behavioral  = true             # requires [dist.build]; set false for artifact-only runs

Adapter contract (what the runner needs from a distribution):

1. An SPDX SBOM file (or the ability to point at one).
2. Public headers + libs (for static API diff), or an exported-symbol list.
3. (behavioral only) A way to build & invoke the showcase CLI for this dist.

Everything else — golden data, policy, check logic, report generation — ships in
the TCK package and is identical for every vendor. This keeps the per-vendor
surface minimal, exactly like Jakarta's porting package.

---

6. How it plugs into reference-integration CI

We extend the existing tag/release flow (test_and_docs.yml) so that on a tag
(e.g. v0.10.0) we build and publish the TCK binary alongside today's test
reports + build-tools SBOM.

6.1 New Bazel targets (in root BUILD, next to //:sbom)

• //tck:golden_sbom — alias/re-export of //:sbom output for packaging.
• //tck:api_baselines — a rule/script that captures per-module exported-symbol
  & header baselines from the reference build into reference/api-baseline/.
• //tck:requirements_map — generates reference/requirements.json by scanning
  platform_integration_tests/** for @add_test_properties(partially_verifies=...)
  (single source of truth = the tests themselves).
• //tck:score_tck_pkg — a tar/package rule assembling the layout in §4 into
  bazel-bin/score-tck-<version>.tar.gz.

6.2 Workflow steps (add to the tag-gated job in test_and_docs.yml)

      - name: Build s-core TCK package
        run: |
          bazel build --lockfile_mode=error //tck:score_tck_pkg
      - name: Attach s-core TCK to GitHub Release
        uses: softprops/action-gh-release@v2.5.0
        if: github.ref_type == 'tag'
        with:
          files: bazel-bin/score-tck-*.tar.gz
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

This mirrors the existing Upload release asset (attach ZIP to GitHub Release)
step, so no new publishing machinery is needed. The version string comes from
the git tag.

Note on lockfile: any change that adds a Bazel dep for the packaging rule must
refresh MODULE.bazel.lock (see .github/instructions/module-bazel-lock.instructions.md).
Packaging rules here are expected to be dependency-free (tar + scripts), so no
new bazel_dep should be required.

---

7. Vendor usage flow (end user)

# 1. Download the TCK for the platform version they target
curl -LO https://github.com/eclipse-score/reference_integration/releases/download/v0.10.0/score-tck-0.10.0.tar.gz
tar xzf score-tck-0.10.0.tar.gz && cd score-tck-0.10.0

# 2. Describe their distribution (the "porting package")
cp examples/tck.config.example.toml my-dist.toml   # edit paths / build cmd

# 3. Run the kit
./bin/score-tck run --config my-dist.toml \
    --checks sbom,api_static,behavioral \
    --report ./report

# 4. Read the verdict
cat ./report/summary.md        # human-readable
cat ./report/report.json       # machine-readable (CI-gateable)

Exit code is 0 only if all required checks for the declared profile pass, so
a vendor can gate their own release pipeline on it.

---

8. Report format & verdict

• report.json: { dist, tck_version, platform_version, checks: [ {id, family, severity, status: pass|fail|warn|skip, details[], evidence} ], verdict }
• summary.md / report.html: per-family tables (SBOM / API / Behavioral),
  each row linked to a requirement ID where applicable.
• Verdict: CONFORMANT | NON-CONFORMANT | CONFORMANT-WITH-WARNINGS.
  Driven by severity: any required check failing → NON-CONFORMANT.

The JSON is the contract for automation; the HTML/MD is for humans and for
attaching to a distribution's release notes as conformance evidence.

---

9. Phased roadmap

Phase 0 — Foundations (design sign-off)

• Agree conformance criteria & which checks are required per profile (§3).
• Finalize tck.config schema (§5) and report schema (§8).

Phase 1 — SBOM check (highest value, no toolchain needed)

• Golden SBOM packaging (//tck:golden_sbom).
• sbom_check.py: presence, required-components, forbidden-components, license
  policy, drift. Run fully offline against a vendor SPDX file.
• Ship first TCK binary + release attachment; document in USER_GUIDE.md.

Phase 2 — Static API check

• //tck:api_baselines capture from the reference build (per module).
• api_check.py: symbol/header diff vs baseline (missing/renamed = fail, added
  = warn).

Phase 3 — Behavioral check (reuse PITs)

• //tck:requirements_map generated from platform_integration_tests/**.
• behavioral_check.py: build & run the showcase CLI per [dist.build], map
  results to requirement IDs. This is the strongest conformance signal and
  reuses existing tests verbatim.

Phase 4 — Hardening & breadth

• Multi-arch (aarch64, QNX, eBCLFSA) profiles.
• Optional feature_integration_tests (ITF) as a deeper functional profile.
• Signed TCK artifacts + pinned golden-data hashes for trust.
• "Certification" policy layer on top of the report (who signs off, appeals —
  mirroring Jakarta's test-appeals process).

---

10. Open questions (need your input)

1. What exactly is a "distribution"? A Bazel workspace they build themselves,
   prebuilt libs+headers, or an OS/container image? This decides how much of the
   [dist.build] adapter we must support in v1. (I've assumed any of these via
   the manifest; confirm.)
2. Required vs informational checks for v1. Is SBOM presence + license policy
   the only required gate initially, with API/behavioral as warnings? Or should
   behavioral (PITs) be required from day one?
3. API baseline source. Do we already have a stable machine-readable public-API
   export (symbols/headers) per module to diff against, or should "API compatible"
   be defined purely by "the platform integration tests pass"? (Affects Phase 2
   scope.)
4. Toolchain assumption for behavioral checks. Are vendors expected to have a
   working SCORE toolchain to build/run showcases, or do we need the TCK to bundle
   a hermetic toolchain? (Bundling is heavier but more turnkey.)
5. Licensing of the TCK & shipped golden data. The golden SBOM contains component
   hashes/versions — confirm that's acceptable to ship publicly in a release asset.
6. Target arches for v1 — confirm linux-x86_64 only, matching current CI.

11. Phase 1 Draft Implementation Status

✅ Done

Component	File	Description
TCK Runner	tck/runner/src/main.rs	CLI with run, validate-config, version commands
SBOM Check	tck/runner/src/sbom.rs	Required components, license policy, drift detection
Version Check	tck/runner/src/version.rs	Platform version compatibility (semver)
Runtime Check	tck/runner/src/runtime.rs	Toolchain detection, test harness adapter concept
Golden SBOM	tck/reference/sbom.spdx.json	Reference SBOM (162 packages)
Requirements Policy	tck/reference/policy/requirements.toml	Mandatory components, allowed licenses, version reqs
Test Definitions	tck/tests/behavioral.toml	Maps to existing feature_integration_tests/
API Definitions	tck/tests/api_signatures.toml	Required header definitions
Bazel Target	tck/BUILD	//tck:score-tck builds static binary
CI Workflow	.github/workflows/build_tck.yml	Build on PR, bundle on release
Documentation	tck/README.md	Quick start, architecture, check details

❌ Not Done (Gap to Production TCK)

Gap	What's Missing	Effort
Behavioral Test Runner	TCK doesn't invoke pytest - test definitions exist but no execution code	~2 days
API Signature Validation	Headers defined but no code validates them against headers_dir	~1 day
Showcase Execution	TestHarnessAdapter has run_showcase() but no integration with actual showcase CLI	~1 day
Bundle Verification	No SHA256 hash in release notes, no smoke test of bundled artifact	~0.5 day
User Guide	No standalone user guide (like Jakarta TCK Users Guide)	~1 day
Sample Distribution Test	Never tested TCK against a real third-party distribution	~1 day

---

---

Appendix A — Existing assets we reuse (reference)

Asset	Path / target	Role in TCK
Product SBOM (SPDX)	//:sbom → bazel-bin/sbom.spdx.json	Golden SBOM for comparison
Tracked modules list	known_good.json → sbom.tracked_modules	"Required components" set
Platform integration tests	platform_integration_tests/**	Behavioral + API conformance engine
Requirement IDs	@add_test_properties(partially_verifies=["stkh_req__*"])	Conformance criteria mapping
Showcase CLI (Rust)	//showcases/bin/cli (showcases/cli/main.py→main.rs)	Driven by behavioral check
Release publishing	.github/workflows/test_and_docs.yml + softprops/action-gh-release@v2.5.0	Attach TCK binary on tag
Build config	--config=linux-x86_64 (.bazelrc)	v1 target platform

[Copilot]

🟡 Changes recommended

Critical validation flaws allow incompatible distributions to be reported as conformant, while CI and packaging defects block reliable verification and release assembly.

18 open issues

• Literal ${pwd} breaks AArch64 sysroot resolution · New
• Smoke test checks the wrong SBOM path and hides failures · New
• Runner file blocks creation of the bundle directory · New
• Released bundles omit the required policy file · New
• TOML comments are mishandled and configured values are lost · New
• Unsupported check selections silently produce success · New
• Compiler versions are not checked against minimums · New
• Missing golden SBOMs are replaced with empty baselines · New
• Substring OR bypasses SPDX license policy validation · New
• Required license violations are incorrectly reported as warnings · New
• Missing SPDX document structure is accepted as conformant · New
• Name-only package keys discard duplicate package entries · New
• SBOM policy sections are never loaded from requirements.toml · New
• Unknown version policies disable enforcement · New
• Invalid platform versions pass as required warnings · New
• Version parsing drops prerelease and invalid components · New
• Configured enabled checks are ignored · New
• Drift checks ignore version and checksum changes · New

What changed in this PR

Introduces the Phase 1 SCORE Test Compatibility Kit within reference integration, with a Rust runner for distribution checks and compatibility reports. Behavioral and API coverage remain definition-only groundwork.

Changes:

• Adds SBOM, version, and runtime checks with JSON and Markdown reporting.
• Adds conformance policy, example configuration, documentation, and build/release packaging.
• Adds Linux AArch64 toolchain configuration and ELF compatibility testing, alongside an image-rule override and lockfile updates.

File	Description
tests/​toolchain/​linux_aarch64_elf_compatibility_test.sh	Checks payload architecture, GLIBC requirements, and segment alignment.
tests/​toolchain/​BUILD	Registers the AArch64 compatibility test.
tck/​tests/​behavioral.toml	Defines behavioral test mappings.
tck/​tests/​api_signatures.toml	Defines required and forbidden headers.
tck/​tck.config.example.toml	Provides a vendor configuration example.
tck/​runner/​src/​version.rs	Implements platform-version checks.
tck/​runner/​src/​sbom.rs	Implements component, license, and baseline comparisons.
tck/​runner/​src/​runtime.rs	Adds environment checks and adapter scaffolding.
tck/​runner/​src/​report.rs	Generates JSON and Markdown reports.
tck/​runner/​src/​main.rs	Adds CLI commands and check orchestration.
tck/​runner/​src/​lib.rs	Exposes reusable runner modules.
tck/​runner/​src/​config.rs	Loads distribution configuration.
tck/​runner/​Cargo.toml	Declares Rust dependencies and release settings.
tck/​reference/​policy/​requirements.toml	Defines platform conformance policy.
tck/​README.md	Documents usage, checks, and implementation scope.
tck/​BUILD	Defines runner, unit-test, and reference targets.
MODULE.bazel.lock	Updates resolved dependencies and toolchain records.
bazel_common/​score_images.MODULE.bazel	Pins an image-rule revision for overlay support.
bazel_common/​score_gcc_toolchains.MODULE.bazel	Adds the Linux AArch64 GCC toolchain.
.github/​workflows/​build_tck.yml	Adds build, smoke-test, bundle, and release steps.
.bazelrc	Adds Linux AArch64 build settings.

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended...

Addressed the review findings in 968d286; workflow token permissions were also restricted in e60eb22.

[github-actions]

The created documentation from the pull request is available at: docu-html