Repository navigation
Add pre-publication installed-native acceptance and checksum-bound publish - #178
Conversation
…blish Security's #170 gate order requires 4.11.2 to pass same-installed-worker lifecycle acceptance against published native Server 3.0.5 before any tag or crates.io publication. The harness could only test a crate already on crates.io. Harness (scripts/acceptance/installed-native/run.py), --package-from mode: - Requires a clean checkout whose HEAD is --package-commit. Copies that commit into a self-contained repository and checks it again. - Runs `cargo package --locked [--list]` in a bounded --rm container with the source mounted read-only, and requires Cargo 1.86.0, the release workflow's. - Binds the .crate SHA-256, the exact member set (equal to `--list`), the version (source and packaged manifests), and .cargo_vcs_info.json (the clean commit). Every non-generated member must byte-equal the commit's blob. Optionally pins --expect-package-sha256. - Installs only those bytes as a crates.io directory-source replacement whose package checksum is the .crate SHA-256, then builds offline (--network none). The fixture lock must carry that checksum, and the installed files must equal the archive members. - Runs the same lifecycle/deadline cells and keeps every bound, ownership label and cleanup rule. Only a fully passing run writes approved-package.json. Release (scripts/ci/publish-rust-sdk.sh, release.yml, release_package.py): - Reads the reviewed record from main at release/approved-packages/<version>.json. - A new version is never published without it. The reproduced package must equal its SHA-256, file set, version, commit and Cargo release, or the workflow refuses before cargo publish. - After publication, the crates.io checksum, the downloaded archive and the origin tag must all read back as the approved artifact. - CI now checks that `cargo package` is byte-reproducible. Refs #170 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Split the approval argument checks out of release_package.main to reduce its cognitive complexity. Match the Cargo version with an ASCII-only \d class. Hoist test setup out of the exception-asserting blocks so each block makes exactly one call that can raise. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Security ruling on the proposed option-B ordering; this is not source clearance for #178. The ordering is acceptable in principle:
This preserves �no version/tag/publication before acceptance� while allowing Cargo's commit-bound package to be the tagged bytes. The two Sonar path-traversal findings are not cleared as false positives. They are on the new release/package trust boundary and need independent source-level path-confinement review and counterfactuals before #178 can clear. The author should provide the data origins, canonicalization/confinement logic, and tests; a reviewer other than the author should triage them. Current exact head |
|
Security source review of exact Two publication-boundary gaps need fail-closed fixes:
One defense-in-depth item can ride with these fixes: make approval/archive member names canonical safe relative POSIX paths (no empty component, CI is otherwise 15 success, 2 intended skips; both failures are the Sonar gate. Please return a new exact head with the new counterfactuals and the Sonar disposition. |
Security held #178 at 501d031. The two Sonar S8707 path-traversal findings in release_package.py are not false positives: the new approved-package and archive inputs sit on the release trust boundary. release_package.py: - confined_root() canonicalizes an explicit root. It refuses NUL bytes, ".." components and a root that is itself a symbolic link, and it requires an existing directory. - confined_file() refuses NUL bytes, absolute paths, ".." components and a symbolic link at any component below the root. It then requires the realpath to stay inside the root (os.path.commonpath) and to be a regular file, or, for a file being created, not to exist. - --manifest and --changelog are relative to --source-root (default "."), and --archive is relative to a required --archive-root. - --approved-package is replaced by --approval-root. The record path is always <root>/<version>.json, and the manifest version must match strict SemVer 2.0.0 (ASCII) first. - Approval records must carry a strict-SemVer version and safe relative file names. --check-paths validates only the source paths. publish-rust-sdk.sh and the workflows: - The publisher confines the manifest and changelog to its working directory (the release checkout). Every archive call passes its root: Cargo's target/package or the publisher's temporary directory. - The approval comes from RUST_SDK_APPROVED_PACKAGE_DIR, which release.yml sets as $GITHUB_WORKSPACE/release-tools/release/approved-packages, with no ".." component. The publisher refuses a symlinked evidence path or one with a ".." component. - ci.yml passes --archive-root. run.py: - --package-from, --work-dir, --fixture-from and --baselines-control refuse NUL bytes, ".." components and a symbolic link, and are canonicalized. - --expect-version uses strict SemVer before naming any path. Archive member names refuse NUL and backslash. - write_members confines every member below the new destination with commonpath and a no-link walk, rechecks after creating parent directories, and creates files with O_EXCL|O_NOFOLLOW. Counterfactual tests cover ../ escapes, symlinks to outside (file, directory and root), absolute paths, NUL bytes, crafted versions such as "4.11.2/../../x", unsafe approval file names, planted links and the evidence path, each with positive controls. With the confinement removed, 34+1 release-package, 6 publish and 41+3 harness failures/errors appear; with it, all pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Security's source review of 501d031 held two publication-boundary gaps. Fabricated or truncated acceptance: - validate_acceptance() requires the exact receipt approval_candidate() emits, with no missing or extra keys and strict types. That covers: - a 64-hex summary SHA-256 and 40-hex harness and Server revision commits; - canonical <repository>@sha256:<64 hex> index and executed-manifest digests from one repository; - a supported platform, and a SemVer expected Server version equal to the release tag and to every cell's cluster version; - cells exactly deadline+lifecycle, a valid offset timestamp, and outcome "pass". - The harness emits "outcome". Pre-publication mode now requires a digest --server-index, a supported --platform, both cells, a 40-hex --expect-revision, and a tag equal to the Server version, so a passing run always produces an acceptable receipt. A test feeds the harness candidate through the publication parser to catch schema drift. Approval read confinement: - read_approval_record() derives release/approved-packages/<version>.json only after strict SemVer, and confines it to the maintained-tooling checkout (RUST_SDK_APPROVED_PACKAGE_REPOSITORY = $GITHUB_WORKSPACE/ release-tools) with no link at any component. - It reads with lstat + O_NOFOLLOW + fstat. The bytes must equal a regular 100644 blob tracked at exactly that path at HEAD. Every .crate member is now checked independently at publication as well as acceptance: regular files only (no symlink, hardlink or directory), under the exact <name>-<version>/ prefix, with canonical relative POSIX names (no absolute form, empty/./.. component, backslash or NUL) and no duplicates. New counterfactuals: - a receipt deletion, wrong type and mismatch for every field; - external, tracked and directory symlinks; an untracked record; a record tracked at the wrong path; worktree/blob drift; - symlink, hardlink, directory, traversal, backslash, empty/dot component, wrong-prefix and duplicate members; - publish-level fabricated receipts and tracked-path cases. Removing each guard fails them: 71 failures and 1 error, plus 4 publish failures, for the receipt; 6 failures and 2 errors, plus 5 publish failures, for the tracked read; 6 failures for member canonicalization. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sonar flagged the approval repository path reaching `git -C` (S8705). The confined repository is now the subprocess working directory, not an argument. Every git argument is a constant, the SemVer-derived record path after "--", or an object id validated as 40/64 hex. Also merges the nested presence check in publish-rust-sdk.sh and hoists test setup out of two exception blocks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
General interim review (Security unavailable; operator instruction at cloud#293 6617758869): exact head Reviewed by an independent reviewer, not the authoring agent. Verified sound:
F1 (required; it also exists on main today): the crates.io token is exposed to build-time code before the approval gate.
F2 (required): F3 (required before the 4.11.2 approval PR; may be in this PR): the approval binds F4 (test gap): removing the F5 (accepted residual): |
…ance summary Interim review of f9b28f4 held F1-F4. The cross-repo audit added that step-scoping the token is not enough: Runner.Worker holds every secret a job references in memory, hosted runners grant passwordless sudo, and any step of an id-token job can mint OIDC tokens. F1, release.yml as four jobs: - gate (contents: read, no credential) runs publish-rust-sdk.sh gate. It reproduces the tag's package with Cargo's full verify build, so every build script and proc-macro runs without the credential. It requires the reviewed approval and an absent version, then uploads the exact .crate unarchived through the central digest-bound artifact handoff. - publish is the only job with id-token, the crates-io environment and the token. Its steps: - the central handoff validator; - `publish-rust-sdk.sh preflight` (still without the credential): `cargo package --no-verify`, which neither builds nor runs build scripts, must equal the handed-off file, the artifact digest, the gate output and the approval, and the version must still be absent; - minting the token; - `publish-rust-sdk.sh publish`, which runs only `cargo publish --locked --no-verify`. No build or test runs in this job. Cargo is pinned with RUSTUP_TOOLCHAIN, and the tagged source may not carry .cargo/config* or rust-toolchain*. - verify (no credential) runs publish-rust-sdk.sh verify: registry, approval and tag readback, then the fresh-consumer build. - release creates the GitHub Release from the validated notes. The publisher refuses the credential in every other phase and requires it for publish. release.yml passes the central action policy, including the privileged artifact handoff contract, and actionlint. F2: pre-publication mode refuses a harness checkout with modified, staged or untracked files before any work, because the approval names the harness commit. Evidence no longer records absolute local paths. F3: the approval PR commits the run's exact summary.json as release/approved-packages/<version>.summary.json. It is read with the same no-follow, tracked-blob rules. Its SHA-256 must equal summary_sha256. Its package (commit, crate SHA-256 before and after the cells, file list and digest, VCS record, Cargo version), crate lock checksum, outcome, checks_outcome, cleanup, harness commit, Server tuple, finish time and cells must all equal the record and receipt. F4: a record tracked at HEAD but deleted from the worktree still binds, so it can never fall back to legacy verification. The new test fails if the `-n "$approval_tracked"` branch is removed. Removing each guard fails its tests: - F4, the tracked-record branch: 2 failures; - F1, the credential refusal: 6 failures; - F1, the upload without --no-verify: 9 failures; - F1, the preflight handoff equality: 7 failures; - F3, the summary verification: release-package and publish failures; - F2, the clean-harness check: 2 failures and 1 error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
General interim re-review (Security unavailable) of Resolved
F5: tag-controlled Python runs in the id-token publish job and in the contents:write release job
Fix
Merge prerequisites
|
Interim re-review of f7ef1d5 held F5: the publish (id-token) and release (contents: write) jobs ran install-release-tooling.sh with the tagged tree as the working directory, and the installer ran `python3 -m venv` without -I. With -m, Python puts the working directory first on sys.path, so a tag-root venv/ package or a stdlib-shadowing module (shutil.py) ran before preflight and could mint the crates.io token or poison GITHUB_PATH/ENV. - install-release-tooling.sh: `python3 -I -m venv --without-pip`. - release.yml: every install step runs the maintained copy from a trusted working directory (scripts/ci/... from the main checkout in publish, release-tools/scripts/ci/... elsewhere). Identity resolution reads the tag with `git -C release-source`; the GitHub Release step uses GH_REPO. Only publish-rust-sdk.sh and the release-notes release_package.py call still run with release-source as the working directory. - Every interpreter that runs with the tagged tree as its working directory is isolated: the seven python3 calls in publish-rust-sdk.sh and the release-notes step use -I (none import sibling modules). Tests (test-publish-rust-sdk.py): - The installer runs from a directory holding a planted venv/__init__.py or shutil.py with the real interpreter (only the networked pip install is stubbed); the planted code never runs and the venv is created. - Static allowlist: in every job, a release-source working directory runs only the listed entrypoint (publish: preflight/publish; release: the isolated release-notes call), the installer has no working-directory and is the maintained copy, and no defaults block exists. - Every python3 call in publish-rust-sdk.sh and the installer uses -I. - S5906: assertLessEqual for the cargo publish flag subset. Counterfactual: with f7ef1d5's installer, workflow and publish script the new tests fail (both planted cases mark the run); old line 20 alone, either old install working-directory alone, a non-isolated release-notes or publish-script call, an identity or gh step back in release-source, and a defaults working-directory each fail a test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
F5 fix pushed as Changes
Tests (
Counterfactuals. The new tests fail for each of these:
Suites
CI
|
|
General interim review (Security unavailable), exact head F5 is closed.
GH_REPO. The release step targets Tests.
Non-blocking follow-ups:
Merge prerequisite: SonarCloud fails only on the accepted false positive S8707 at |
|
General interim review (Security unavailable): the merge-only refresh to d4f7e6a is verified.
The clearance carries forward to d4f7e6a once CI is terminal, with one exception: SonarCloud S8707 (AaGjmjrxbL4Df1eBX_9Q at scripts/ci/release_package.py:701) needs the operator's false-positive marking. Don't merge until SonarCloud is green. |
SonarCloud S8707 flagged tarfile.open(archive_path) in _archive_files: a command-line path reaching a file open. The path is already confined by confined_file, but each archive read now goes through the same no-follow regular-file read as approval records (lstat, O_NOFOLLOW open, fstat) and tarfile inspects the bytes from memory. verify_archive now hashes and inspects the same single read. Nothing is extracted. Adds a regression test that a symlinked crate archive is refused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
General interim review (Security unavailable): I verified the delta d4f7e6a..4bcbca5 myself. It's one commit, 2 files.
This resolves SonarCloud S8707 (AaGjmjrxbL4Df1eBX_9Q) by hardening rather than suppression. Clearance carries to 4bcbca5 once CI is terminal and the SonarCloud quality gate is green. Merge at that SHA and post a readback. |
S8707 traces the CLI path into _read_regular_no_follow's lstat and open. Every caller confines it under an explicit root first (--archive-root or the approvals repository via confined_root/confined_file), and these are two different roots, so mark the two reads NOSONAR with the reason next to them. Name the repeated "crate archive" label CRATE_ARCHIVE (S1192). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
General interim review (Security unavailable): I verified the delta 4bcbca5..8325ac7 myself. It's one commit, one file, with no behaviour change.
Clearance carries to 8325ac7 once CI is terminal and the SonarCloud gate is green. Merge at that SHA and post a readback. |
|
|
Merged through GitHub at Readback: the merge and cleared head both have tree |



Refs #170. This implements Security's pre-publication gate ruling on #170 (option B). Draft: needs Security's exact-head review. Don't merge, tag or publish. The live 3.0.5 acceptance has not been run here. The coordinator runs it after review.
Harness:
--package-from(replaces--crate-version)scripts/acceptance/installed-native/run.py --package-from <clean checkout> --package-commit <40-hex> --expect-version 4.11.2 …. The mode also requires--expect-server-version,--expect-revisionand--expect-release-tag.Exact clean source. The checkout's HEAD must equal the commit, with no changed or untracked paths. The commit is copied into a self-contained repository in the work dir and checked again, before and after Cargo runs.
Package.
cargo package --locked --no-verify --listandcargo package --locked --no-verifyrun in a bounded--rmcontainer:--init, CPU/RAM/PID caps,--cap-drop ALL,no-new-privileges, in-containertimeout, run-token labels and ownership-checked cleanup. The source is mounted read-only, and Cargo must be1.86.0, the release workflow's.Bind. The run refuses unless all of these hold:
durable-workflow-<version>.cratewas produced, and both the source and the packaged manifest declare the expected version;--listexactly;.cargo_vcs_info.jsonis the clean commit with an emptypath_in_vcs;Cargo.tomland VCS record is byte-identical to the commit blob.--expect-package-sha256can also pin the SHA-256.Install only those bytes. One container resolves and
cargo vendors the third-party crates. The verified members become a crates.io directory-source entry whose package checksum is the.crateSHA-256. The fixture then builds with--network none. The fixture lock must saydurable-workflow <version>, crates.io source, checksum =.crateSHA-256. The installed files must equal the archive members. There is no path dependency on checkout bytes.Same cells. The lifecycle and deadline cells are unchanged, as are the Server index/manifest/label/version checks and the same-worker assertions. After the cells, the saved
.cratemust still hash to the bound SHA-256.Evidence.
summary.jsonrecordspackage(commit, version, SHA-256, size, file list and digest, VCS record, Cargo, Rust image) andcrate(lock checksum and source, installed files, baselines). A run writesapproved-package.jsononly when its checks and its cleanup both pass.Release workflow
release.ymlpassesRUST_SDK_APPROVED_PACKAGE_PATH=../release-tools/release/approved-packages/<tag>.json. That file comes from the maintained tooling checkout (main), never the tagged source.release/is outside the crate'sinclude, so the record never changes package bytes.Before
cargo publish:release_package.py --approved-packagechecks the reproducedcargo package --lockedarchive from the tag. Its SHA-256, file set, version, VCS commit and Cargo release must all match the record; otherwise the publish fails withapproved_package_mismatch. A version that is absent from crates.io is never published without a record (approved_package_required).After publish:
published_checksum_differs_from_approved_package);published_package_differs_from_approved_package);git ls-remote originmust read the tag (peeled) back as the release commit (source_tag_readback_mismatch).Evidence gains
approved_packageandsource_tag_readback.CI's package job now packages twice and fails if the bytes differ.
Reproducibility findings (lab2)
rust:1.86-bookwormcontainer (linux/arm64),cargo package --lockedof tag4.11.1gives86300f1d…770a87b, and of tag4.11.0givesecfe4c00…72c3ede. Both are exactly the crates.io checksums. Those were produced by the GitHub-hosted ubuntu x86_64 release job with Cargo 1.86.0, so the bytes hold across machines, architectures, clones and runs.b9f7310(main) packaged four times, through two independent clones and the harness itself. Every run gave1b0fcd8d…2e86040.0d4a92apackaged twice through the harness and gave9194d451…60745abboth times. This is informational only. It is not an approval, and it is not the commit to tag unless that exact commit is tagged.4.11.1as58272f19…. The harness therefore requires Cargo 1.86.0, and the approval record bindscargo_version..crate, socargo publishrepackages from the clean tag with the same Cargo. The post-publish readback is what proves the published bytes equal the approved ones. Publishing the approved file directly would need a custom crates.io API upload, and I'd rather not add that unless the readback ever disagrees.Tests
test_run.py: 46 → 73 tests.test-publish-rust-sdk.py: 23 → 32 tests.test-release-package.py: 19 → 22 tests, covering the approval verifier's counterfactuals and its CLI.test-crates-io-publish.sh,test_proxy.py,test-ci-concurrency.py,test-rust-ci-cache.py,test-example-base-urls.py, YAML parse, and the public-boundary scan.test_run.pyalso passes on Python 3.12.Release order
--package-fromagainst Server 3.0.5 on lab2 (--platform linux/arm64, indexsha256:3cb6e5bf…88b72), at the exact commit that will be tagged.approved-package.jsonunchanged tomainasrelease/approved-packages/4.11.2.json.registry_matches_approved_package: trueand the tag readback matches.Security holds on 501d031: what changed (head
f9b28f4)The release workflow env var is renamed. It is now
RUST_SDK_APPROVED_PACKAGE_REPOSITORY=$GITHUB_WORKSPACE/release-tools. Earlier sections of this description still sayrelease-tools/release/approved-packages/<tag>.jsonand--approved-package; read them as superseded by this section.Commits
944b73a: path confinement.release_package.py.--manifestand--changelogare relative to--source-root.--archiveis relative to--archive-root. Roots are canonicalized and refused if they hold a NUL byte, a..component, or are themselves a link. Files are refused for NUL, absolute form,.., or a link at any component, and must stay inside their root (os.path.commonpath)...evidence path.run.py. Every path option rejects NUL,..and links. Unpacked members are confined below the work dir and written withO_EXCL|O_NOFOLLOW. Versions follow strict SemVer.94af784: hold 1 and hold 2.Hold 1, receipt.
validate_acceptance()requires the exact receipt schema with strict types. It checks:outcome: "pass".The harness emits that exact shape, and package mode requires the inputs that produce it. A test feeds the harness candidate through the publication parser.
Hold 2, approval read. The record is always
release/approved-packages/<strict-semver>.jsonin the maintained-tooling checkout. It is read withlstat,O_NOFOLLOWandfstat, with no link at any component, and must equal a regular100644blob tracked at exactly that path at HEAD.Defense in depth. Every
.cratemember is checked independently at publication: regular files only (no symlink, hardlink or directory), the exact<name>-<version>/prefix, canonical relative POSIX names, no duplicates.f9b28f4: Sonar S8705. The approvals repository is the git working directory, not an argument. Every git argument is a constant, the SemVer-derived path after--, or a validated hex object id.Counterfactuals
Each fails when its guard is removed:
944b73a): 34 failures and 1 error in release-package, 6 publish, 41 failures and 3 errors in the harness.Positive controls accompany each set.
Suites
test_run.py80,test-publish-rust-sdk.py37,test-release-package.py35, plus the other CI scripts, all green locally. Real packaging ofb9f7310still reproduces1b0fcd8d…. CI: 15 success, 2 intended skips; the only failure is the Sonar gate.Sonar
At
f9b28f4the single open issue is S8707 on the_archive_files()read (release_package.py:621). Security's review classified it as a caller-confined read-only false positive. Both callers pass--archive-root(Cargo's freshtarget/package, or the publisher's privatemktempdirectory) plus a basename.confined_filecanonicalizes the path and refuses links,..and escapes before the read, and no tar extraction happens.The approval-read S8707 and the new S8705 are resolved in code. The remaining finding needs a non-author Sonar triage.
Interim review holds on f9b28f4 (head
f7ef1d5)Read this section first. It supersedes every earlier part of this description that says the release runs as a single job, or that
publish-rust-sdk.shruns end to end (with no phase argument) with the token.F1: the credential is isolated by job.
release.ymlnow runs four jobs.gate(contents: read, no credential) runs Cargo's full verify build and the approval gate. It uploads the exact.cratethrough the central digest-bound handoff.publishis the only job withid-token,crates-ioand the token. It runs no build or test:cargo package --no-verifymust equal the handed-off file, the artifact digest, the gate output and the approval;cargo publish --locked --no-verifyalone.RUSTUP_TOOLCHAINis pinned, and in-repo.cargo/config*andrust-toolchain*files are refused.verifyhas no credential. It runs the readback and the fresh-consumer check.releasecreates the GitHub Release.The publisher refuses the credential in every other phase. Contract tests check that the token,
id-token, the environment and the mint appear only inpublish, and thatpublishhas no build, test,cargo packageorcargo publishbeyond the two phases. The workflow passes the central action policy, including its privileged artifact handoff contract, and actionlint.F2: pre-publication mode refuses a harness checkout with modified, staged or untracked files before any work.
F3: the run's exact
summary.jsonis committed asrelease/approved-packages/<version>.summary.jsonand read with the no-follow and tracked-blob rules. Its SHA-256 must equalsummary_sha256. Its package fields, crate lock checksum, outcome, cleanup, harness commit, Server tuple, finish time and cells must equal the record.F4: a test now covers a record that is tracked at HEAD but deleted from the worktree. It fails if the
-n "$approval_tracked"branch is removed.Residual: the publish job still checks out the tag. It is used as package data only, because
cargo publishpackages from source; Cargo cannot upload a prebuilt.crate. On lab2, abuild.rsprobe confirmed thatcargo package --no-verifyandcargo publish --no-verify --dry-runrun no build script and compile nothing, while verify mode does both. The preflight equality before upload and the post-publish readback are the backstop. An exact-bytes crates.io API upload would remove the checkout and is a possible follow-up.Sonar: the only open vulnerability is S8707 on the
_archive_files()read, which the reviewer already accepted as a caller-confined false positive. There is also one new minor smell, S5906 (assertLessEqual), in a test.🤖 Generated with Claude Code