Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 42 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: ["3.11", "3.12"]
python-version: ["3.11", "3.12", "3.13"]

steps:
- uses: actions/checkout@v4
Expand All @@ -25,3 +25,44 @@ jobs:

- name: Run test suite (CPU-only)
run: python -m pytest tests/ -q

coverage:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- name: Install package with dev extras
run: pip install -e ".[dev]"
- name: Enforce 100% line and branch coverage
run: |
coverage run -m pytest tests/ -q
coverage report --fail-under=100
coverage xml
- uses: actions/upload-artifact@v4
with:
name: coverage-xml
path: coverage.xml

docs-and-package:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- name: Install package with dev extras
run: pip install -e ".[dev]"
- name: Build documentation strictly
run: mkdocs build --strict
- name: Build and inspect distributions
run: |
python -m build
python -m twine check --strict dist/*
- name: Smoke-test the wheel
run: |
python -m venv /tmp/wheeltest
/tmp/wheeltest/bin/pip install -q dist/*.whl
/tmp/wheeltest/bin/gpu-proof --help
/tmp/wheeltest/bin/python -m pytest -p pytest_gpu_proof --version
29 changes: 29 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# Agent guide

This repository is a security-sensitive pytest plugin. Preserve fail-closed
behavior and treat receipt, fingerprint, signature, Git, merge, and policy code
as trust boundaries.

Before changing behavior, read:

- `README.md`
- `docs/architecture.md`
- `docs/security_model.md`
- `docs/policy.md`
- `CONTRIBUTING.md`

Implementation rules:

- New receipts are schema 3; keep explicit legacy verification isolated.
- Signer metadata belongs inside the signed payload.
- Default fingerprint scope is the full tracked repository; generated inputs
require explicit extra paths.
- Never turn receipt-emission failures into success unless the caller selected
`best_effort`.
- Never accept unknown policy fields or malformed receipt structure.
- Keep tests hermetic: mock GitHub key lookup and local identity.
- Maintain 100% line and branch coverage. Add adversarial cases for new trust
branches.
- Do not edit generated `site/` or release artifacts in `dist/`.

Run the complete check list from `CONTRIBUTING.md` before proposing a PR.
63 changes: 63 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,69 @@ All notable changes to pytest-gpu-proof are documented here. The format is
based on [Keep a Changelog](https://keepachangelog.com/); versions follow
[SemVer](https://semver.org/) (pre-1.0: minor bumps may break).

## [Unreleased] — 0.4.0

### Added

- Schema `"3"`: signer username, key fingerprint, key algorithm, exact test
collection, session outcome, pytest arguments, and per-shard environment/time
are included in the signed payload.
- Open contributor and restricted signer policies. Restricted mode supports
username and exact SSH-key-fingerprint allowlists.
- Policy pinning for mode, global tracked/extra fingerprint scope, exact test
manifest, shard names, and per-shard tracked/extra scope.
- Explicit fingerprint paths for ignored/generated inputs and manifest support
for symlinks and submodule gitlinks.
- Explicit receipt-artifact exclusions avoid self-referential whole-tree
fingerprints and can be pinned by verification policy.
- `--gpu-proof-best-effort` as an explicit development escape hatch.
- `min_schema` policy field to refuse legacy schema-1/2 receipts.
- Python 3.13 CI, strict docs/package jobs, and enforced 100% line and branch
coverage.

### Changed

- The default fingerprint scope is the entire Git-tracked repository instead
of `src,tests`.
- Receipt creation now fails pytest on Git, fingerprint, key, serialization, or
write errors; it also removes stale output at session start.
- Schema-3 verification rejects dirty recording and verification trees by
default and accepts receipt commits only at the current commit or an ancestor.
- Setup/teardown failures, missing terminal reports, comparison exceptions,
skipped tests, and overall session failure are represented and verified.
- GPU metadata records all devices reported by `nvidia-smi`.
- Array comparison is shape-safe, uses tolerance only for float/complex data,
and uses exact equality for other dtypes.
- Receipt writes are atomic and strict JSON forbids NaN/non-JSON values.
- Receipt generation rejects xdist workers; use separate shard processes.
- Legacy schema-1/2 verification derives the policy-checked key fingerprint
from the key that actually verified the signature; an asserted
`signature.key_fingerprint` that disagrees is rejected.
- `signer_mode: restricted` policies reject unsigned receipts even with
`--allow-unsigned`, and `--github-user` must match the signed schema-3
identity.
- The receipt under verification is excluded from the verification-tree dirty
check, so an untracked just-generated receipt verifies without gitignoring.
- Passphrase-protected SSH keys prompt correctly on current cryptography
releases (`ValueError` as well as `TypeError`) and fail closed with an
actionable message when no terminal is available.
- Uninitialized submodule checkouts fingerprint their index gitlink commit
instead of accidentally recording the parent repository's HEAD.
- Receipt/shard age limits are enforced to the exact day boundary.
- GitHub usernames are validated before key fetches and key responses are
size-capped.
- A stale receipt that cannot be cleared at session start raises a pytest
usage error (an exit-status write that early would be silently overwritten).
- Merging receipts without session timestamps is refused with a clear error.

### Fixed

- Carry-forward now recomputes the stored fingerprint algorithm and preserves
explicit generated/ignored shard inputs.
- Signed identity and algorithm substitution are rejected.
- Empty source scope, unmerged index entries, malformed policies, incomplete
collections, and stale success artifacts fail closed.

## [0.3.0] — 2026-08-08

### Added
Expand Down
81 changes: 11 additions & 70 deletions CLAUDE.md
Original file line number Diff line number Diff line change
@@ -1,73 +1,14 @@
# CLAUDE.md — orientation for AI agents (and humans) working on pytest-gpu-proof
# Claude orientation

pytest-gpu-proof is a **pytest plugin that emits a signed JSON receipt of a GPU
test run** (git SHA + source fingerprint + per-test outcomes + GPU info), plus a
**CPU-only verifier** (`gpu-proof verify`) that checks the signature against the
signer's public `github.com/<user>.keys`. The point: run GPU tests on your own
hardware, let CI prove who attested them — no cloud-GPU fees, no secrets in CI.
Use [AGENTS.md](AGENTS.md) as the canonical agent guide. In particular:

**Trust model — never oversell it:** a receipt is a **signed attestation by a
keyholder, NOT cryptographic proof of GPU execution**. `gpu_info` is
self-reported; `--require-gpu` is modest hardening against accidents, not
against a dishonest signer. `docs/security_model.md` is the honest statement of
what is and isn't proven — keep every README/docs claim consistent with it.
- a receipt is signer attestation, not proof of GPU execution;
- receipt, fingerprint, signature, Git, merge, and policy code fail closed;
- new recordings use schema 3 and bind signer metadata inside the signature;
- the default manifest covers the full tracked repository;
- tests are hermetic and maintain 100% line and branch coverage;
- generated `site/`, `dist/`, coverage output, real keys, and local receipts are
not source changes.

## Source layout (`src/pytest_gpu_proof/`)

| Module | Role |
|---|---|
| `plugin.py` | pytest hooks: options, `gpu_proof` marker, `gpu_proof_check` fixture, outcome+skip capture, receipt emission at session end |
| `receipt.py` | payload build (repo/fingerprint/tests/env), canonical JSON, sign+write. Signer resolution: flag/config → `gh` CLI login (keyholder) → origin-remote owner (warned — orgs have no SSH keys) |
| `verify.py` | the 7 verification checks (signature, fingerprint, commit SHA, outcomes+skip policy, gpu_info, freshness, dirty policy). Expected-skips baseline = EXACT set match |
| `cli.py` | `gpu-proof verify` argument surface |
| `config.py` | `GpuProofConfig`; CLI flags override `[tool.gpu_proof]` in pyproject.toml |
| `fingerprint.py` | SHA-256 digest over configured paths |
| `gitutils.py` | git/gh shell-outs, all failure-tolerant (return `None`) |
| `signers/` | `base.py` protocol + `ed25519.py` SSH-key signing / GitHub-keys verification; backend `none` emits unsigned receipts |
| `compare.py` | receipt diffing |

Signing covers the canonical (compact, sorted-key) JSON **without** the
`signature` field; the sig block carries `signer`, key fingerprint, algorithm
(derived from actual key type — don't hardcode ed25519).

## Behavioral invariants (test-enforced — don't regress)

- **Skips prove nothing.** Verifier rejects receipts with skips unless
`--allow-skipped` (any skips) or `--expected-skips` (EXACT baseline: a new
skip fails, a stale baseline entry fails). The two are mutually exclusive.
- **Unsigned receipts** verify only with `--allow-unsigned`, loudly.
- **`--max-age-days 0`** means "today only", not "disabled".
- The signer recorded at signing time must be the **keyholder**, never
silently the repo owner.

## Dev workflow

```bash
.venv/bin/python -m pytest tests/ -q # full suite, ~2s, no GPU needed
.venv/bin/mkdocs build --strict # docs must stay warning-clean
```

- Tests are **hermetic**: `tests/conftest.py` has an autouse fixture nulling
`get_gh_cli_login` (a dev box with authenticated `gh` would otherwise hit the
network). Signer tests re-patch explicitly. Keep new shell-outs mockable and
wrapped in try/except like `gitutils._git`.
- `tests/conftest.py` uses `pytest_plugins = ["pytester"]`; keypairs are
generated in-memory (no real SSH keys touched).
- requires-python ≥ 3.11 (`datetime.UTC`).
- CI (`.github/workflows/`): tests on 3.11/3.12 + mkdocs gh-pages deploy.

## Conventions & state

- Short single-line commit messages; no Co-Authored-By footer.
- Flow: feature branch → PR → CI green → merge to `main`. Consumers install
from git (`pip install -e` on a submodule) or PyPI. **Releases**: OIDC
trusted publishing via `.github/workflows/publish.yml` (TestPyPI on manual
dispatch, PyPI on GitHub release) — process in `RELEASING.md`; keep
`CHANGELOG.md` current and bump `pyproject.toml` version in the same PR.
- Reference integration: **GLASS** (github.com/A2R-Lab/GLASS) —
`test/run_gpu_proof.sh`, `test/expected_skips.txt`,
`.github/workflows/verify-gpu-proof.yml`. If you change plugin/verifier
flags, check GLASS's usage still works and note it in the PR.
- Consumer-side gotcha worth remembering: a repo that submodules this project
under its pytest rootdir must `collect_ignore = ["pytest-gpu-proof"]` in its
conftest, or this repo's `tests/conftest.py` will shadow theirs.
Before implementation work, read `docs/architecture.md`,
`docs/security_model.md`, `docs/policy.md`, and `CONTRIBUTING.md`.
41 changes: 41 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# Contributing

## Development setup

```bash
python -m venv .venv
source .venv/bin/activate
python -m pip install -e ".[dev]"
```

## Required checks

```bash
python -m pytest tests -q
coverage run -m pytest tests -q
coverage report --fail-under=100
mkdocs build --strict
python -m build
python -m twine check --strict dist/*
```

CI runs the test suite on Python 3.11, 3.12, and 3.13, independently enforces
100% line and branch coverage, builds docs strictly, and smoke-tests the wheel.

## Change expectations

- Add adversarial tests for trust-boundary changes, not only happy paths.
- Keep receipt generation and verification fail-closed by default.
- Treat schema and policy changes as compatibility decisions; document them in
`CHANGELOG.md` and the relevant guide.
- Keep Git and network tests hermetic. Never depend on a developer's actual
GitHub login, SSH keys, GPU, or global Git configuration.
- Use separate pytest processes for shards. Receipt generation does not support
xdist workers.
- Do not commit generated `site/`, coverage files, caches, or local receipts.

## Pull requests

Describe the trust claim before and after the change, migration implications,
and the exact checks run. Small, reviewable commits are preferred. Security
reports should follow [SECURITY.md](SECURITY.md), not a public issue.
Loading
Loading