From 1ab4c89315f458de6d7a52835aedd3807cb7aff4 Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Mon, 26 Oct 2026 16:06:12 -0700 Subject: [PATCH] Host the Hardened Secret and Env Readers in the Hub and Carry Them as a Tree (#3509) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Implements #3505, with the design points the maintainer settled on the issue before the first edit. ## What Changed - **`lib/sh/secrets.sh`** is the hub's runtime library: `unresolved_link_reason`, `secret_state`, the shape, content, set, and full verdicts, `require_secret`, and `read_env_value`. Each function body is byte-identical to the source implementation, compared with `declare -f`. Its comments now describe the mechanism rather than the adopter. The installer path helpers, the push interlock, and `write_state_file` stay in the source repository. The source has no owner check, so the issue's owner-exemption parameter had nothing to parameterize. - **`host-secrets`** is a new project type. `spec/project-types.json` gives it `detect` and one intent check, `spec/scope-model.md` lists its token, `spec/type-model.md` gains "Runtime Libraries", and the conformance matrix has its row. - **`spec/files.json`** carries `lib/sh` to `lib/fleet/sh` as a `verbatim-tree` for `appliesTo: ["host-secrets"]`, so `carry.py`, the audit, and the canonical sweep pick it up with no code of their own. - **`repo_gate.py` `carried-trees`** compares each carried tree a scanned repository tracks with the hub tree beside the gate, the pinned action's in CI. It loads the hub's own `scripts/carry.py` and reuses its tree functions, including the new `target_patterns` and `compare_declaration` that `carry.py`'s own check and post-apply comparison now share. A difference is a `note:` plus a `::warning` annotation and never a finding. A repository tracking `spec/files.json` is a hub and compares nothing. A tree that cannot be compared, or a `carry.py` that will not load or run, says so in a note. - **`GOVERNANCE.md`**: "Hub-Hosted Tooling" states why code a repository's own runtime executes is carried rather than reached. "Repo-Scoped Secrets" states the reader rule the type's check judges, and "Repository Layout" lists `lib/`. - **Tests**: `tests/test_secrets_sh.py` drives every verdict and the env read against constructed files. `tests/test_repo_gate.py` covers the new check and `gate_hub_root`. `tests/test_cli_usage.py` lists the library as sourced rather than run. ## Verification - The full unit suite under `tests/requirements.txt` passes: 3023 tests, 7 skipped. - Mutating the library, by dropping `-u BASH_ENV` or the `unset`, reading `$2` at the end, dropping `/dev/null`, reordering the set verdict, or sourcing in process, fails the case written for each fault. - `spec/validate.py`, `build_dist.py --check`, the prose gate with comments allowed, every `repo_gate.py` check, and the Docker lint wrapper's seven linters are clean. - `carried-trees` was run against two downstream checkouts. Blog's `.github/skills` matches the hub. One checkout's copy differs in 37 files and warns as designed. The check also runs under Python 3.12, the runner's system interpreter. ## Local Review Each push's content went through recorded `local-strict-review` passes at the `frontier` tier, and each push spent its two-round edit budget. The seven pushes' passes raised 11, 15, and 4 findings, then 6 and 4, then 4, 4, and 3, then 5 and 3, then 2 and 2, then 3 and 1, then 3 and 1, and the eighth push's pass raised none. The seventh push also fixes the Windows unit test job, which failed on an unresolved 8.3 temp path in the new `carry.run` case. Every `introduced` finding is answered. ### Declined in the sixth push - The import-time `uv python find` probe in `tests/host_capability.py` follows the file's existing eager `can_symlink()` probe. ### Pre-existing findings, not filed - `pre-existing`: `tests/test_repo_gate.py`, whose write-verb scan reads `scripts/repo_gate.py`, a `run_path` shim, rather than the gate under `.github/actions/repo-gate/`. ### Declined, with evidence - The library's own hardening (the positional-parameter guard, `grep -D skip`, the `readarray -d /` split): the bodies move byte-identical by the issue's design, so a change to them is its own issue in the authoring repository. - The note not naming the hub commit: the pinned action's tree is an archive with no `.git`, and the run log already names the action's pin. - Untracked files in a local hub checkout's source tree: `carry.py`'s own `inventory` reads the source the same way, so the gate and a carry agree. - CRLF working-tree bytes on a Windows checkout: every registry entry declares `lineEndings: lf`. - The `spec/files.json` hub test catching an unpruned template copy: the note says "is a hub", so the result is visible rather than silent. - The issue's point that the same rule gives `hub_fetch_run.py` a declared home: that loader is not a `spec/files.json` tree, and the "Hub-Hosted Tooling" paragraph beginning "A loader is outside this section" already governs it. - The `try` around the manifest read: `load_json` belongs to a `carry.py` of whatever hub version the pin holds, so the reason for the other broad catches covers it too. - Folding the gate test helpers `TestCarriedTrees.write` and `TestGateHubRoot.place` into one: a preference between defensible forms, which "Review the Change" says to omit. ### Dropped findings - The module docstring's table framing, and long sentences in comments and docstrings, which an advisory check reports and no gate enforces. - `spec/audit.py` `check_tree_verbatim` keeps its own blob-based statement of the tree ownership rule. That file is untouched here, and it reads blobs rather than inventories. The pull request carries the `comments` label. The added comments are each function's caller contract and the measured reasons behind each check. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .github/actions/repo-gate/action.yml | 2 +- .github/actions/repo-gate/repo_gate.py | 139 +++++++++++ .github/workflows/validate-task.yml | 2 +- AUDIT.md | 1 + GOVERNANCE.md | 4 + lib/sh/secrets.sh | 243 +++++++++++++++++++ reports/conformance-matrix.md | 1 + scripts/README.md | 3 +- scripts/carry.py | 33 ++- spec/fidelity-model.md | 2 +- spec/files.json | 3 +- spec/project-types.json | 6 + spec/scope-model.md | 2 +- spec/type-model.md | 5 + tests/host_capability.py | 22 ++ tests/test_carry.py | 52 ++++ tests/test_cli_usage.py | 5 +- tests/test_repo_gate.py | 304 +++++++++++++++++++++++ tests/test_secrets_sh.py | 322 +++++++++++++++++++++++++ 19 files changed, 1136 insertions(+), 15 deletions(-) create mode 100644 lib/sh/secrets.sh create mode 100755 tests/test_secrets_sh.py diff --git a/.github/actions/repo-gate/action.yml b/.github/actions/repo-gate/action.yml index 549717439..64372e25c 100644 --- a/.github/actions/repo-gate/action.yml +++ b/.github/actions/repo-gate/action.yml @@ -1,5 +1,5 @@ name: Repository gate -description: Check action pins, line-ending contracts, and composite action shell and schema. +description: Check action pins, line-ending contracts, composite action shell and schema, and carried tree currency. inputs: exclude-globs: diff --git a/.github/actions/repo-gate/repo_gate.py b/.github/actions/repo-gate/repo_gate.py index 832bd76b2..3177b6c3e 100755 --- a/.github/actions/repo-gate/repo_gate.py +++ b/.github/actions/repo-gate/repo_gate.py @@ -8,6 +8,7 @@ eol .editorconfig <-> .gitattributes disagreement 40 findings eol-coverage Git attribute resolution differs from policy count not recorded composite-actions shellcheck and schema coverage of action.yml count not recorded + carried-trees a manifest-owned tree differing from the hub's count not recorded `eol-coverage` asks Git how representative paths resolve. This proves the global text default reaches Python, shell, Dockerfiles, workflow YAML, and extensionless scripts without maintaining @@ -35,12 +36,15 @@ from __future__ import annotations import argparse +import importlib.util import json +import os import re import shutil import subprocess import sys from pathlib import Path +from types import ModuleType # GOVERNANCE.md documents exactly one floating-ref exception. SHA_EXCEPTIONS = {"dotnet/nbgv"} @@ -769,11 +773,146 @@ def check_composite_actions(root: Path, files: list[str]) -> list[str]: return hits +def gate_hub_root() -> Path | None: + """The hub tree this gate file sits in, or None where it was lifted out of one. + + The pinned action runs from the whole hub tree at its pin, and so does a run from a hub + checkout. A single-file fetch has no tree around it, and whatever happens to sit three levels + above the file is not the hub, so the path is trusted only where its own parents name it. + """ + here = Path(__file__).resolve() + if [part.name for part in here.parents[:3]] != ["repo-gate", "actions", ".github"]: + return None + root = here.parents[3] + return root if (root / "spec" / "files.json").is_file() else None + + +def hub_carry(hub: Path) -> ModuleType: + """The hub's own `scripts/carry.py`, whose tree inventory and comparison this check reuses. + + It imports only the standard library until a section is read, so plain `python3` loads it. + """ + spec = importlib.util.spec_from_file_location("carry", hub / "scripts" / "carry.py") + assert spec is not None and spec.loader is not None + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def warn_carried_tree(message: str) -> None: + """Show `message` on the pull request under Actions, built from manifest values alone so nothing a scanned repository names reaches the command.""" + if os.environ.get("GITHUB_ACTIONS") == "true": + print(f"::warning title=carried-trees::{message}") + + +def check_carried_trees(root: Path, files: list[str]) -> list[str]: + """Report each manifest-owned tree the scanned repository carries that differs from the hub's. + + `spec/files.json` declares the trees a repository carries byte-for-byte, and a carried copy + goes stale silently at the next hub fix. Comparing against the hub tree this gate runs from + makes the answer reproducible at the action's pin, so a Dependabot bump of that pin is what + surfaces a refresh the resync then applies. + + A difference is a note rather than a finding, so it never fails the gate. Failing would break + every downstream pull request on each hub fix, and a byte difference cannot tell a stale copy + from an edited one, which the audit tells apart. Which trees a repository owes is the audit's + too, read from its declared types, so a tree absent here is not compared. The copy is read + from its tracked files alone, so an untracked file is not a difference. + """ + hub = gate_hub_root() + if hub is None: + NOTES.append( + "no hub tree is beside this gate, as with a single-file fetch, " + "so no carried tree was compared." + ) + return [] + # A hub tracks its manifest, which no carrying repository does, so a hub worktree on another branch is not read as a stale copy. + if "spec/files.json" in files: + NOTES.append("the scanned repository is a hub, whose trees are sources rather than copies.") + return [] + # Each failure is a note rather than a traceback, since one here would also skip every check sorted after this one. + # The catches are broad because carry.py runs under whatever `python3` the host has, so a newer interpreter's API in it fails here as any exception. + try: + carry = hub_carry(hub) + except Exception as error: # noqa: BLE001 + NOTES.append( + f"the hub's scripts/carry.py could not be loaded, so no carried tree was compared: {error}" + ) + return [] + try: + trees = carry.load_json(hub / "spec" / "files.json")["trees"] + except Exception as error: # noqa: BLE001 + NOTES.append(f"no carried tree was compared: {error}") + return [] + carried = 0 + for tree in trees: + target = tree["target"] + prefix = target + "/" + # Git tracks a link to a directory as one path, so a target or an ancestor tracked that way hides every file beneath it. + # A top-level ancestor such as `lib` is generic, so one tracked that way says nothing about a carry and is passed over like an uncarried tree. + blocking = next( + ( + path + for path in files + if (path == target or "/" in path) and prefix.startswith(path + "/") + ), + None, + ) + if blocking is not None: + warn_carried_tree( + f"{target} sits at or below a tracked link or file, which a carry refuses" + ) + NOTES.append( + f"{target} was not compared, since {blocking} is tracked as a link or a file." + ) + continue + tracked_here = [path[len(prefix) :] for path in files if path.startswith(prefix)] + if not tracked_here: + continue + try: + owned = [p for p in tracked_here if carry.included(p, carry.target_patterns(tree))] + # A tree that does not prune shares its target with the repository's own files, so those alone do not make it carried. + if not owned: + continue + # A carry refuses a link anywhere under a tree's target, so the gate refuses one too rather than judging it by a rule of its own. + link = next((p for p in sorted(tracked_here) if (root / target / p).is_symlink()), None) + if link is not None: + warn_carried_tree(f"{target} holds a link, which a carry refuses") + NOTES.append( + f"{target} could not be compared, since {link} is a link, which a carry refuses." + ) + continue + # A tracked file absent from the working tree is left out, so it reads as missing. + copy = { + relative: (root / target / relative).read_bytes() + for relative in owned + if (root / target / relative).is_file() + } + source = carry.inventory(hub / tree["source"], tree["include"]) + result = carry.compare_declaration(tree, source, carry.Inventory(copy, frozenset(), "")) + except Exception as error: # noqa: BLE001 + NOTES.append(f"{target} could not be compared: {error}") + continue + carried += 1 + changed, missing, extra = result["modified"], result["missing"], result["extra"] + if changed or missing or extra: + warn_carried_tree(f"{target} differs from the hub's {tree['source']}") + NOTES.append( + f"{target} differs from the hub's {tree['source']}: {len(changed)} changed, " + f"{len(missing)} missing, {len(extra)} extra, first " + f"{(changed + missing + extra)[0]}. Refresh it from the hub, or let the audit say " + "whether it is a stale copy or an edited one." + ) + NOTES.append(f"compared {carried} carried tree(s) against the hub tree beside this gate.") + return [] + + CHECKS = { "sha-pin": check_sha_pin, "eol": check_eol, "eol-coverage": check_eol_coverage, "composite-actions": check_composite_actions, + "carried-trees": check_carried_trees, } diff --git a/.github/workflows/validate-task.yml b/.github/workflows/validate-task.yml index 6a53138cf..79fed87a9 100644 --- a/.github/workflows/validate-task.yml +++ b/.github/workflows/validate-task.yml @@ -23,7 +23,7 @@ on: default: '' # Threaded to the repo-gate composite action's own exclude-globs input, unvalidated. repo-gate-exclude-globs: - description: Git pathspec patterns to exclude, one per line (for example 'themes/PaperMod/**'), narrowing the sha-pin, eol-coverage, and composite-actions scans via repo_gate.py's --exclude. + description: Git pathspec patterns to exclude, one per line (for example 'themes/PaperMod/**'), narrowing the sha-pin, eol-coverage, composite-actions, and carried-trees scans via repo_gate.py's --exclude. required: false type: string default: '' diff --git a/AUDIT.md b/AUDIT.md index 953a6290e..9e18a647e 100644 --- a/AUDIT.md +++ b/AUDIT.md @@ -58,6 +58,7 @@ Otherwise read its `types[]`. If the entry is `classificationPending` (a backlog - `Dockerfile` + a docker build/push workflow -> `docker`, an `upstream-version.json` tracker -> `upstream-wrapper`. - `custom_components/*/manifest.json` + `hacs.json` -> `homeassistant`, a codegen workflow -> `codegen`, no `build-*` task -> `source-only`, governance-only -> `docs`. - `hugo.yaml` / `hugo.toml` / `config/_default/hugo.yaml` -> `hugo`. A repo may carry it alongside `source-only`, since a site deploy leaf is not a `build-*` task and both declarations stay true. +- a carried `lib/fleet/sh/secrets.sh` -> `host-secrets`, and so does a shell script the repo runs that reads a credential file under `.secrets/` or `~/.secrets/`. ## 3. Applicability Gate diff --git a/GOVERNANCE.md b/GOVERNANCE.md index 923ef5f04..c8596b41f 100644 --- a/GOVERNANCE.md +++ b/GOVERNANCE.md @@ -123,6 +123,7 @@ A repo whose own stacks or scripts read local runtime credentials from disk, mos A fresh checkout then documents its own required shape without ever exposing a real value. This negation keeps a real secret file out of a **new** commit. It does not remove one already tracked: `.gitignore` has no effect on a path git already follows. A real secret file found tracked is removed from the index (`git rm --cached `) and its credential is rotated, not just added to `.gitignore` going forward. - **`.secrets/README.md` is a catalog**, one row per file naming what it holds and what consumes it, plus a short note on how the directory relates to `~/.secrets/` where the repo also touches that. +- **A repository whose shell scripts read a credential from disk declares the `host-secrets` type, and reads each one through the carried readers.** A credential is a secret file or an env file holding one, in `.secrets/` or in the host-scoped `~/.secrets/`, and this bullet reaches both. A single inline read counts, since it is the read a bare existence check and a sourced file get wrong. A script sources its carried `lib/fleet/sh/secrets.sh`, per "Hub-Hosted Tooling", and checks the file before reading it. A private file takes `require_secret` or `secret_set_verdict`, and one a container reads as a Docker secret takes `secret_verdict` with `shared`. It reads one value from an env file with `read_env_value` rather than sourcing the file into its own shell. ## Repository Onboarding and Conformance @@ -139,6 +140,8 @@ The fleet's tooling lives in the hub once and a repository runs it from there ra **What a repository carries and what it reaches is decided by what the content is.** It carries the content it is audited against and the configuration that describes it, meaning its rule text and the files the manifest declares. It reaches machinery whose content is identical in every repository, meaning the prose and repository gates, the review digest, and the configuration script, because a file holding no per-repo content is a copy whose only future is to go stale. A tool named in a carried rule is therefore named as the hub's, since the alternative is a pointer to a path the reader does not have, and a pointer that resolves nowhere teaches the reader that a pointer in carried text is decorative. +**Code a repository's own runtime executes is the exception, and it is carried.** Such a library runs on a host the project owns, often unattended and sometimes as root. Fetched from the hub at run time, it would let a GitHub outage or a bad `main` commit stop a scheduled host job. It would also run code the repository never reviewed against its own secrets. So offline operation and review in the repository that runs it outweigh the staleness of a copy. A gate differs, since "the tool did not run" is an acceptable answer there and CI is the backstop. The hub hosts the library once. A repository whose declared types select it carries it byte-for-byte, as a tree the hub's `spec/files.json` declares. The hub owns the carried directory whole. The repository extends the library in a file of its own outside it that sources it, never by editing the copy. The staleness is made visible rather than accepted. The repository gate compares each carried tree with the hub tree its pinned action holds. It reports a difference as a warning that never fails the gate. A failure would break every downstream pull request on each hub fix. The audit reports a stale copy, and the resync refreshes it. + **Reaching it is a checkout of the hub rather than a copy of one file.** A tool reads the payloads, tables, and sibling modules beside it, so a single file lifted out of the tree runs against whatever the caller happens to have, which is the copy problem again in a shorter loop. Read `main`, the promoted and gated state, and fetch immediately before running, because a clone is whatever it last fetched rather than the branch it names, and a stale clone answers confidently instead of failing. Name the tool by its path in that checkout and name the target explicitly, since a tool that defaults to the current directory or the current repository resolves somewhere either way, and a result computed against the wrong repository is well-formed. Which directory the command runs in is the tool's own contract rather than a rule here, so a gate reading a working tree runs in the repository under test while a tool taking its target as an argument runs anywhere and is given one. What the rule forbids is letting a default decide which repository the answer is about. **A loader is outside this section rather than exempt from it.** The rule above governs a tool that reads hub content, because a tool reads the payloads, tables, and sibling modules beside it. A loader reads none of them: it obtains a tree and hands control to a tool inside that tree, on a host that cannot yet obtain one. The bound is what it may contain rather than who runs it, and it is one line: a loader references no path inside the tree it fetches except the single entry point it hands control to, and everything else it touches is the machine or the network. A loader that grows a second path into that tree has become a tool and is governed above. @@ -374,6 +377,7 @@ Every repo's GitHub repository details (the About panel) follow a fixed conventi - [`.github/skills/`](./.github/skills/): generated in the hub from `.agents/skills/` and carried verbatim to every repo, so a fix to a Skill belongs at that source rather than in any copy of this tree. - [`.claude-plugin/fleet-skills/`](./.claude-plugin/fleet-skills/): hub-only, the same Skills as a Claude plugin. The `marketplace.json` beside it names the plugin rather than the Skills, is hand-authored, and `build_dist.py` never writes it. - [`scripts/`](./scripts/): the fleet's lint and review tooling, with the deterministic checks also gating CI, for the documented rules no CI linter checks. Hosted here and reached rather than carried, per "Hub-Hosted Tooling". +- [`lib/`](./lib/): runtime libraries the hub hosts and a repository carries byte-for-byte, per "Hub-Hosted Tooling", each under `lib/fleet/` in the repository whose declared types select it. `lib/sh/` is the shell secret and env readers, carried to a `host-secrets` repository. - [`tests/`](./tests/): hub-only, the unit tests for the hub's own tooling, which the shared validator runs under coverage as the hub's declared Python directory. - [`docs/`](./docs/): hub-only procedure and measurement docs (host setup, repository configuration, signing, agent token cost), none of them carried downstream. - [`host-setup/`](./host-setup/): hub-only, the installers that bring a machine up to the host contract `docs/host-setup.md` states, one tree each for Linux and Windows, with the cross-platform agent-safety install beside them. diff --git a/lib/sh/secrets.sh b/lib/sh/secrets.sh new file mode 100644 index 000000000..e48a5a0e7 --- /dev/null +++ b/lib/sh/secrets.sh @@ -0,0 +1,243 @@ +#!/bin/bash + +# Meant to be sourced, never run: `. "$REPO/lib/fleet/sh/secrets.sh"` from a script that has already found its own tree through `BASH_SOURCE`. +# It sets no shell option, since sourcing it would change the caller's own, and every function answers through its output or its status under any of them. +# +# The hub hosts this file and a repository carries it byte-for-byte, per GOVERNANCE.md "Hub-Hosted Tooling", so a repository extends it in a file of its own outside the carried directory that sources it, never by editing it. +# A caller that needs these functions in an elevated shell ships them there by name with `declare -f`, since sourcing this file in the calling shell does not reach one started by `sudo`. +# A function here that calls another is shipped with it, so `secret_verdict` needs `secret_shape_verdict`, `secret_state`, `unresolved_link_reason`, `secret_content_verdict`, `secret_content_state`, and `secret_content_refusal`. + +# `unresolved_link_reason` +# Echoes the clause that says why a symlink answering `-L` true and `-e` false does not resolve, worded for the account running it, so every site that reports such a link composes one sentence instead of spelling its own. +# Root with its DAC override enters every directory, so a link it cannot resolve has a target that is missing or loops back on itself, and the clause says only that. +# A uid 0 without the override, such as one in a user namespace facing an unmapped owner, is read as root all the same. +# Any other account cannot tell those two from a target sitting behind a directory it cannot enter, and the clause names all three. +# A caller that tells the two wordings apart does so by the ending ` sits behind a directory this account cannot enter`, fixed text here. +unresolved_link_reason() { + if [[ $EUID -eq 0 ]]; then + echo "its target is missing or loops back on itself" + else + echo "its target is missing, loops back on itself, or sits behind a directory this account cannot enter" + fi +} + +# `secret_state` +# One word for what a secret file is, because a gate that skips on a bare `-f` reads four unrelated states as one, since `-f` answers false for each of them. +# `absent` only where nothing is at the path and every directory leading to it is one this account can actually enter, which is the single state a gate may skip on. +# `damaged ` where the path or a directory leading to it is a symlink that does not resolve, or is present in a shape a secret cannot be read from. +# `unknown ` where a directory leading to it cannot be entered, or the path is not absolute, so the path's absence was never established rather than observed. +# `present` where a regular file is there. +# The walk starts at the root rather than at the immediate parent, since a single unenterable ancestor makes every component below it answer false to `-e` and that would otherwise read as a proven absence. +# See `secret_verdict` below for the full check built on it. +secret_state() { + local path="$1" walked="" part index last + if [[ $path != /* ]]; then + echo "unknown $path is not an absolute path" + return 0 + fi + # A doubled or trailing slash names the same component again, so both are removed before the split rather than becoming empty components the walk would have to skip. + while [[ $path == *//* ]]; do + path="${path//\/\//\/}" + done + path="${path%/}" + # The trim above empties a path that was only slashes, which would otherwise walk no components at all and fall through to `present` without ever reaching the leaf test. + # The root is a directory, so it gets the same answer any other non-regular-file leaf gets rather than a fifth state. + if [[ -z $path ]]; then + echo "damaged / is not a regular file" + return 0 + fi + local -a parts + # Split on `/` alone, since `read` stops at the first newline and a path component may hold one, and a split that fails answers rather than walking no components. + readarray -d / -t parts < <(printf '%s' "${path#/}") || { + echo "unknown $path could not be split into its components" + return 0 + } + last=$((${#parts[@]} - 1)) + for index in "${!parts[@]}"; do + part="${parts[index]}" + walked="$walked/$part" + # `-L` before `-e` throughout: a symlink that does not resolve answers false to `-e`, which is exactly the state a bare existence test reads as an absence. + # Reported as unresolved rather than as a missing target, since `-e` cannot separate a target that is gone from one that loops back on itself or, for a non-root account, one sitting behind a directory this account cannot enter, and asserting any one of them would claim more than the test established. + if [[ -L $walked && ! -e $walked ]]; then + echo "damaged $walked is a symlink that does not resolve, so $(unresolved_link_reason)" + return 0 + fi + if [[ ! -e $walked ]]; then + echo absent + return 0 + fi + if [[ $index -eq $last ]]; then + if [[ ! -f $walked ]]; then + echo "damaged $walked is not a regular file" + return 0 + fi + echo present + return 0 + fi + if [[ ! -d $walked ]]; then + echo "damaged $walked is not a directory, so it can hold no secret" + return 0 + fi + if [[ ! -x $walked ]]; then + echo "unknown $walked cannot be entered by this account" + return 0 + fi + done + echo present +} + +# `secret_shape_verdict` [private|shared] +# What `secret_verdict` below answers, from the file's metadata alone, without opening it. +# `readable` where it is a regular file, not a symlink, gives group and other no access, and this account can read it. +# `shared` drops the group and other check alone, for a secret a container reads as a Docker secret, which stays group and other readable by design. +# `absent` where nothing is there, which `secret_state` above establishes rather than infers, so a caller whose secret is optional may skip on it. +# `refused ` for everything else, the sentence naming the path and why, so a caller prints it as the reason rather than composing its own. +# A symlink is refused by name even where it resolves to a private file, since `stat -c %a` reads the link's own 777 rather than its target's mode, and a link is also a second place the file's content can be repointed from. +# A caller running as root can call this one alone and run `secret_content_state` in the shell it reads the file with, so root never opens the file. +secret_shape_verdict() { + local path="$1" access="${2:-private}" state mode + # Refused rather than normalized, since the caller reads the spelling it was given, and the kernel answers a trailing slash on a file with "Not a directory". + if [[ $path == */ ]]; then + printf 'refused %s ends in a slash, so it names a directory rather than a file\n' "$path" + return 0 + fi + state=$(secret_state "$path") + case $state in + present) ;; + absent) + echo absent + return 0 + ;; + *) + printf 'refused %s\n' "${state#* }" + return 0 + ;; + esac + if [[ -L $path ]]; then + printf 'refused %s is a symlink, and must be the file itself\n' "$path" + return 0 + fi + # Validated as octal digits before `(( ))` reads it, since a non-octal value there fails the test the same way a private mode passes it. + if ! mode=$(stat -c '%a' -- "$path") || [[ ! $mode =~ ^[0-7]{1,4}$ ]]; then + printf 'refused the mode of %s could not be read, so whether it gives group or other access cannot be established\n' "$path" + return 0 + fi + if [[ $access != shared ]] && ((8#$mode & 8#077)); then + printf 'refused %s is mode %s, and must give group and other no access\n' "$path" "$mode" + return 0 + fi + if [[ ! -r $path ]]; then + printf 'refused %s is mode %s and owned by %s, so this account cannot read it\n' "$path" "$mode" "$(stat -c '%U' -- "$path")" + return 0 + fi + echo readable +} + +# `secret_content_state` +# `clean` where the file `secret_shape_verdict` passed holds no carriage return, `crlf` where it holds one, and `unchecked` where the check could not run. +# A FIFO or device in place of the file, or a symlink to one, answers `unchecked`, since `-D skip` passes such a file unread. +# Refused rather than left to each reader, since each reader fails on a carriage return differently. +# A sourcing shell under errexit stops on a blank CRLF line, and one without it keeps the carriage return in every value. +secret_content_state() { + # `grep` streams the file rather than holding it, and `-D skip` keeps `grep` itself from blocking on a FIFO swapped in after the shape checks. + local found=0 + LC_ALL=C grep -qF -D skip -- $'\r' "$1" || found=$? + case $found in + 0) echo crlf ;; + 1) + if [[ -f $1 ]]; then + echo clean + else + echo unchecked + fi + ;; + *) echo unchecked ;; + esac +} + +# `secret_content_refusal` +# The `refused ` for a `secret_content_state` other than `clean`, built without opening the file, and any state but `crlf` reads as `unchecked`. +secret_content_refusal() { + if [[ $2 == crlf ]]; then + printf 'refused %s holds a carriage return, most often from CRLF line endings, and must use LF line endings only\n' "$1" + else + printf 'refused the check of %s for a carriage return could not run, so whether it holds one cannot be established\n' "$1" + fi +} + +# `secret_content_verdict` +# `readable` where `secret_content_state` answers `clean`, and its `secret_content_refusal` otherwise. +secret_content_verdict() { + local state + state=$(secret_content_state "$1") + if [[ $state == clean ]]; then + echo readable + else + secret_content_refusal "$1" "$state" + fi +} + +# `secret_verdict` [private|shared] +# One word for whether may be read as a secret: `secret_shape_verdict`'s answer, then `secret_content_verdict`'s where the shape is `readable`. +# A reader in another language runs this function rather than restating its checks, so the rule exists once, here. +secret_verdict() { + local verdict + verdict=$(secret_shape_verdict "$1" "${2:-private}") + if [[ $verdict != readable ]]; then + printf '%s\n' "$verdict" + return 0 + fi + secret_content_verdict "$1" +} + +# `secret_set_verdict` ... +# One answer for a set of secrets read all or none, from `secret_verdict` on each path. +# `readable` where every path is, which an empty list is too. +# `missing ` where any is absent, naming the final component of each absent path in the order given, separated by single spaces. +# `refused ` otherwise, the first refusal in the order given, so the sentence names the first file a reader of the set would stop on. +# A missing file outranks a refusal, since the set cannot be complete whatever the refused file turns out to be. +# Naming the refusal first would send the operator to fix a mode, only for the next run to name the files still missing. +secret_set_verdict() { + local path verdict missing="" refused="" + for path in "$@"; do + verdict=$(secret_verdict "$path") + case $verdict in + readable) ;; + absent) missing="$missing ${path##*/}" ;; + *) [[ -n $refused ]] || refused="$verdict" ;; + esac + done + if [[ -n $missing ]]; then + printf 'missing%s\n' "$missing" + elif [[ -n $refused ]]; then + printf '%s\n' "$refused" + else + echo readable + fi +} + +# `require_secret` +# Succeeds only where `secret_verdict` answers `readable`, and otherwise prints why to stderr and fails, so a secret that must be there is one bare call before the line that sources it. +# A bare call under `set -e` stops the script on that failure, where a call in an `if` or before `||` would not. +require_secret() { + local verdict + verdict=$(secret_verdict "$1") + case $verdict in + readable) return 0 ;; + absent) printf 'ERROR: %s is not there\n' "$1" >&2 ;; + *) printf 'ERROR: %s\n' "${verdict#refused }" >&2 ;; + esac + return 1 +} + +# `read_env_value` +# Prints as the env file named by sets it, and fails where any line of that file fails. +# A bash of its own rather than a subshell, because a caller reading this inside an `if` condition has errexit off in every subshell as well, so only the last line of the file would count. +# The child also keeps `set -a` from exporting that file's credentials into anything this script spawns, and `unset` keeps an operator's already-sourced shell from standing in for the file. +# The name is held in a variable of the child's own rather than read from `$2` at the end, since the file runs with the positional parameters in view and can change them. +# What the file itself prints goes to /dev/null rather than into the value, and `BASH_ENV`, `POSIXLY_CORRECT` and `SHELLOPTS` are dropped so nothing but the file runs ahead of the read. +# It checks nothing about the file, so a caller runs `require_secret` on it first. +read_env_value() { + # shellcheck disable=SC2016 # Every expansion belongs to the child bash. + env -u BASH_ENV -u POSIXLY_CORRECT -u SHELLOPTS bash -Eeuo pipefail -c 'env_read_name=$2; unset -v "$env_read_name"; set -a; source "$1" >/dev/null; set +a; printf "%s" "${!env_read_name:-}"' _ "$1" "$2" +} diff --git a/reports/conformance-matrix.md b/reports/conformance-matrix.md index f02bf30b2..7af653fe9 100644 --- a/reports/conformance-matrix.md +++ b/reports/conformance-matrix.md @@ -20,6 +20,7 @@ The primary shapes are stood up as whole repos. The **composable targets** (`nug | `upstream-wrapper` | - | not-tested | - | Tag from a committed state file, not SemVer2. | | `codegen` | - | not-tested | - | Deterministic matrix over both branches. | | `docs` | ProjectTemplate | not-tested | - | Governance hub, and CI is lint-only. | +| `host-secrets` overlay | - | not-tested | - | Layered onto a base shape, and carries the hub's `lib/sh` secret and env readers at `lib/fleet/sh` byte-for-byte. HomeAutomation-Config is the first adopter, by its own adoption issue. | | `operational` config | - | not-tested | - | `workflowModel: operational`, direct signed commits to `develop`, promotion-PR gate. HomeAutomation-Config was the reference until its 2026-09-24 reclassification to `release`, and no remaining operational repository (ESPHome-Config, HomeAssistant-Config, Vantage-Config) has taken its place yet. | ## Composable Targets diff --git a/scripts/README.md b/scripts/README.md index b3fa46b04..40c8a91aa 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -123,6 +123,7 @@ Deterministic checks: - `eol`: every path pinned LF in [`.gitattributes`][gitattributes] has the matching [`.editorconfig`][editorconfig] override the line-ending rule requires, with EditorConfig brace syntax expanded. One direction only: an `.editorconfig` LF glob with no git pin is legitimate, since `.editorconfig` governs what the editor writes where git enforces a class it must not guess at. - `eol-coverage`: the same pins read against the tree instead. A tracked file opening `#!` that git does not resolve to `eol=lf` is an interpreter line a CRLF checkout breaks, and a pin matching no tracked file is dead unless its block is marked `forward-declared`. - `composite-actions`: every tracked `action.yml` or `action.yaml` under `.github/actions/`, at any depth, is shellchecked and schema-checked, because actionlint reads workflows only. Each non-empty `run:` body of a bash, sh, or unset-shell step goes to shellcheck. Every `${{ }}` expression is replaced by a placeholder first. A finding names the action file and the step. A pwsh or python step is skipped. The file set goes to `check-jsonschema`'s `vendor.github-actions` builtin through `uvx check-jsonschema@latest`, or `pipx run` where `uvx` is absent. Shellcheck runs from the `mirror.gcr.io/koalaman/shellcheck:stable` image, so the check needs Docker. It reads the action through PyYAML, in process where it is importable and through `uvx --with pyyaml` otherwise. Each half reports its own missing tool. Bodies run at warning severity without SC2154, so this is weaker than shellcheck over a `.sh` file. A tree with no action file is skipped with a note. +- `carried-trees`: each tree [`spec/files.json`][files] declares is compared byte-for-byte with the hub tree beside the gate. Only a tree whose target holds tracked files the tree owns is compared. A tree that prunes owns every file under its target, and one that does not owns the files its `include` matches. The hub tree is the pinned action's in CI and the checkout's in a local run. The comparison reuses [`carry.py`](#carrypy)'s tree functions, so `include` and `prune` read as a carry reads them. A tree that cannot be compared says so in a note, such as one whose hub source holds a link. So does a `carry.py` that will not load. A difference is a `note:` naming its counts and first path, plus a `::warning` annotation on Actions. It is never a finding, so a hub fix does not fail every downstream pull request. A byte difference cannot tell a stale copy from an edited one, so the audit tells them apart and the resync applies the refresh. In CI the comparison is with the pin, so a copy refreshed from a newer hub `main` also warns until Dependabot advances the pin. An `--exclude` pattern narrows this scan like the others, so one covering part of a carried target reads the files it hides as missing. Which trees a repository owes is the audit's, read from its declared types, so a tree the repository does not track is not compared. A note and a warning name a target tracked by Git as a link or a file. So does such an ancestor below the top level. Git then lists nothing beneath it. A link anywhere under a carried target is refused in a note and a warning, as a carry refuses it. A single-file fetch has no hub tree beside it, and says so in a note. A repository tracking `spec/files.json` is a hub, so it compares nothing, since its trees are the sources. ```sh python3 scripts/repo_gate.py @@ -153,7 +154,7 @@ The resolution pass is **scoped to the scanned repository's own owner**, which i A pin is a finding only where GitHub **answered** that the commit is absent, meaning a 404 or a 422. An offline host, a missing `gh`, a rate limit and a narrow token all report as unread and leave the pin on its shape, so the gate stays usable with no network instead of failing a correct tree. A 404 is confirmed against the repository itself before it becomes a finding, since an absent commit and a repository the credentials cannot see are the same answer from here, and a repository-scoped CI token is narrower than the fleet. That second read runs only on the failing path, and each distinct pin is read once however often it repeats. -A `note:` line is how a check says it did less than its name. It prints under the check's own result, outside the issue count, and never changes the exit code, because nothing is wrong with the tree when the network is what is missing. It carries one fixed shape every run so a zero is as visible as a count, and it separates a pin under a known other owner from one whose owner could not be compared at all, since a checkout with no readable `origin` skips every pin including this owner's own and reporting those as somebody else's is the same false clean the note was added to prevent. +A `note:` line is how a check says it did less than its name. `carried-trees` also uses one to report a difference it deliberately does not count, for the reason its bullet above gives. It prints under the check's own result, outside the issue count, and never changes the exit code. For a check that read less, that is because nothing is wrong with the tree when the network is what is missing. The `sha-pin` note carries one fixed shape every run so a zero is as visible as a count. It separates a pin under a known other owner from one whose owner could not be compared at all. A checkout with no readable `origin` skips every pin, this owner's own included. Reporting those as somebody else's is the same false clean the note was added to prevent. A stale-backticked-path check was built and **rejected**: a template repo legitimately references paths that live in downstream repos, so it produced 34 false positives on a clean tree with no way to separate those from real drift. Doc-to-doc drift is a review lens, not a regex. diff --git a/scripts/carry.py b/scripts/carry.py index 470185996..90b2f4c7b 100755 --- a/scripts/carry.py +++ b/scripts/carry.py @@ -5,6 +5,9 @@ # /// """Check or apply manifest-owned trees, and the verbatim sections inside a mixed file, from the hub to a fleet worktree.""" +# The repository gate's `carried-trees` check loads this whole file under plain `python3`, 3.12 on a CI runner, and calls its tree functions. +# The file's top level and those functions therefore stay on the standard library and on 3.12, and the gate reports a failure to load or run them as a note. + import argparse import fnmatch import functools @@ -141,6 +144,22 @@ def compare(source: Inventory, target: Inventory | None) -> dict[str, Any]: } +def target_patterns(declaration: dict[str, Any]) -> list[str]: + """What a declaration owns under its target: everything where it prunes, its `include` where it does not.""" + return ["**/*"] if declaration.get("prune") else declaration["include"] + + +def compare_declaration( + declaration: dict[str, Any], source: Inventory, target: Inventory | None +) -> dict[str, Any]: + """`compare`, with the extras cleared where the declaration does not prune, since it owns only what it carries there.""" + result = compare(source, target) + if not declaration.get("prune"): + result["extra"] = [] + result["extraDirectories"] = [] + return result + + def apply_tree( source: Inventory, target_root: pathlib.Path, @@ -890,12 +909,10 @@ def run(mode: str, name: str, target: pathlib.Path, hub: pathlib.Path = ROOT) -> source_root = relative_root(hub, declaration["source"]) target_root = relative_root(target, declaration["target"]) source = inventory(source_root, declaration["include"]) - target_patterns = ["**/*"] if declaration.get("prune") else declaration["include"] - target_inventory = inventory(target_root, target_patterns) if target_root.exists() else None - result = compare(source, target_inventory) - if not declaration.get("prune"): - result["extra"] = [] - result["extraDirectories"] = [] + target_inventory = ( + inventory(target_root, target_patterns(declaration)) if target_root.exists() else None + ) + result = compare_declaration(declaration, source, target_inventory) print( json.dumps( {"source": declaration["source"], "target": declaration["target"], **result}, @@ -919,7 +936,9 @@ def run(mode: str, name: str, target: pathlib.Path, hub: pathlib.Path = ROOT) -> if mode == "apply" and not declaration_clean: for change in apply_tree(source, target_root, target, result): print(change) - final = compare(source, inventory(target_root, target_patterns)) + final = compare_declaration( + declaration, source, inventory(target_root, target_patterns(declaration)) + ) print( json.dumps( { diff --git a/spec/fidelity-model.md b/spec/fidelity-model.md index 920331b31..b9629483e 100644 --- a/spec/fidelity-model.md +++ b/spec/fidelity-model.md @@ -52,7 +52,7 @@ So a local rule edit to `CODESTYLE.md` or `WORKFLOW.md`, both `intent` and whole A verbatim check compares content by hash after **line-ending, action-pin, and job-needs normalization**. EOL variance is governed by the line-ending rules, a `uses: @` pin (with its trailing `# vN` comment) is Dependabot-owned and bumped per repo, and a job's `needs:` list is pruned per repo to its vendored targets (an unvendored name fails the workflow to load), so all three are governed drift rather than a fidelity deviation. This keeps a verbatim workflow job region (the `github-release` job) from flagging on a routine action bump or a legitimate needs-prune while still catching a real structural fork. It does **not** mask placeholders: a verbatim unit carries none. The files that declare a `placeholders` list (for example `.github/copilot-instructions.md` with ``, ``, ``) are fidelity `intent`, judged by hand and never hashed. Masking could not serve a hash anyway. A downstream copy holds the substituted value (`ptr727`), not the token (``), so masking the token in the canonical alone would guarantee a mismatch. A verbatim unit that ever needed a per-repo substitution would require template-matching (the canonical as a pattern, the copy as an instance), not this content hash. None does today. -`verbatim-tree` applies no normalization. It compares file bytes because a fully owned generated tree has no per-repository substitutions or Dependabot-owned regions. +`verbatim-tree` applies no normalization. It compares file bytes because a fully owned tree has no per-repository substitutions or Dependabot-owned regions. ## Stale Versus Modified diff --git a/spec/files.json b/spec/files.json index b5a6d7b8e..e29a3b69a 100644 --- a/spec/files.json +++ b/spec/files.json @@ -2,7 +2,8 @@ "$schema": "./files.schema.json", "note": "The standardization baseline: files, sections, and fully owned trees a fleet repo is expected to carry. See spec/scope-model.md for applicability and spec/fidelity-model.md for fidelity.", "trees": [ - { "source": ".github/skills", "target": ".github/skills", "fidelity": "verbatim-tree", "appliesTo": "*", "include": ["**/*"], "prune": true } + { "source": ".github/skills", "target": ".github/skills", "fidelity": "verbatim-tree", "appliesTo": "*", "include": ["**/*"], "prune": true }, + { "source": "lib/sh", "target": "lib/fleet/sh", "fidelity": "verbatim-tree", "appliesTo": ["host-secrets"], "include": ["**/*"], "prune": true } ], "baseline": [ { "path": "CLAUDE.md", "fidelity": "verbatim", "whole": true, "appliesTo": "*" }, diff --git a/spec/project-types.json b/spec/project-types.json index a593b3008..16454ef83 100644 --- a/spec/project-types.json +++ b/spec/project-types.json @@ -130,6 +130,12 @@ "checks": [ { "id": "docs.ci.lintonly", "verdict": "intent", "assert": "CI runs linting only (markdownlint, cspell, JSON schema, actionlint); no build/test.", "intentRef": "AGENTS.md" } ] + }, + "host-secrets": { + "detect": ["lib/fleet/sh/secrets.sh", "a shell script the repo runs that sources or reads a credential file under .secrets/ or ~/.secrets/"], + "checks": [ + { "id": "hostsecrets.library.reader", "verdict": "intent", "assert": "Every shell script the repo runs that reads a credential from disk, a secret file or an env file holding one under .secrets/ or ~/.secrets/, sources the carried lib/fleet/sh/secrets.sh and checks the file before reading it, with require_secret or secret_set_verdict for a private file, or secret_verdict with shared for one a container reads as a Docker secret, and reads one value from an env file with read_env_value rather than sourcing the file into its own shell. A repo-specific helper lives in the repo's own file outside lib/fleet/sh that sources the library, never in an edit to the carried copy or a file beside it, whose byte identity the carried tree's verbatim-tree fidelity already checks.", "intentRef": "GOVERNANCE.md#repo-scoped-secrets" } + ] } }, "crossCutting": { diff --git a/spec/scope-model.md b/spec/scope-model.md index 62e7bd057..dcde572ae 100644 --- a/spec/scope-model.md +++ b/spec/scope-model.md @@ -15,7 +15,7 @@ A selector is one token from one of four **disjoint** namespaces. Because the na | Namespace | Tokens | Source of truth | | --- | --- | --- | -| project type | `csharp` `nuget` `pypi` `python` `cpp` `dotnet-publish` `docker` `homeassistant` `eda` `codegen` `upstream-wrapper` `source-only` `hugo` `docs` | [`spec/project-types.json`][project-types] | +| project type | `csharp` `nuget` `pypi` `python` `cpp` `dotnet-publish` `docker` `homeassistant` `eda` `codegen` `upstream-wrapper` `source-only` `hugo` `docs` `host-secrets` | [`spec/project-types.json`][project-types] | | workflow model | `release` `operational` | [`registry/repos.schema.json`][repos-schema] | | release trigger | `two-phase` `dispatch-only` `none` | [`registry/repos.schema.json`][repos-schema] | | consumer model | `push` `pull` | [`registry/repos.schema.json`][repos-schema] | diff --git a/spec/type-model.md b/spec/type-model.md index d2adf7ccb..336ded0d9 100644 --- a/spec/type-model.md +++ b/spec/type-model.md @@ -56,6 +56,10 @@ There is no `static-site` to `hugo` hierarchy while the type has one member. Ins A generator type declares no `profiles`. Build versus lint-only is a depth of *language* presence, so a profile on a generator type would assert nothing, and `spec/validate.py` rejects a declared profile whose type does not define one. +## Runtime Libraries + +A **runtime library** type declares that the repo's own runtime executes a library the hub hosts. The type's token selects the library's tree in [`spec/files.json`][files], and the repo carries it byte-for-byte. `host-secrets` is the one such type, for a repo whose shell scripts read a credential from disk. It carries `lib/sh` from the hub to `lib/fleet/sh`. Its `detect` entry surfaces such a script as a discovery advisory, and the declaration is what makes the type's check apply, as for any type. A runtime library type declares no `profiles`, for the reason a generator type declares none. + ## Changing the type set carries review weight The set of types, their profiles, and each type's checks is governed, like the section and fidelity models. @@ -72,6 +76,7 @@ The set of types, their profiles, and each type's checks is governed, like the s [audit]: ./audit.py [fidelity-model]: ./fidelity-model.md +[files]: ./files.json [repos]: ../registry/repos.json [schema]: ./project-types.schema.json [section-model]: ./section-model.md diff --git a/tests/host_capability.py b/tests/host_capability.py index e54041791..5e9fb221d 100644 --- a/tests/host_capability.py +++ b/tests/host_capability.py @@ -35,6 +35,28 @@ def can_symlink() -> bool: requires_symlink = unittest.skipUnless(can_symlink(), NO_SYMLINK) +@functools.cache +def python_3_12_installed() -> bool: + """Whether uv finds a Python 3.12 already installed, so a case using it never downloads one.""" + if shutil.which("uv") is None: + return False + try: + found = subprocess.run( + ["uv", "python", "find", "--no-python-downloads", "3.12"], + capture_output=True, + timeout=60, + check=False, + ) + except (OSError, subprocess.SubprocessError): + return False + return found.returncode == 0 + + +requires_python_3_12 = unittest.skipUnless( + python_3_12_installed(), "no Python 3.12 is installed for uv to run" +) + + @functools.cache def unprivileged_root_is_available() -> bool: """Whether `unshare -r` gives this process an EUID of 0 without any real privilege. diff --git a/tests/test_carry.py b/tests/test_carry.py index a7679713f..1fd97b870 100755 --- a/tests/test_carry.py +++ b/tests/test_carry.py @@ -3,6 +3,9 @@ from __future__ import annotations +import contextlib +import io +import json import pathlib import re import subprocess @@ -42,6 +45,55 @@ def test_compare_reports_missing_modified_extra_and_directories(self) -> None: self.assertEqual(result["missingDirectories"], ["empty"]) self.assertEqual(result["extraDirectories"], ["extra-dir"]) + def test_apply_leaves_a_repository_file_under_a_tree_that_does_not_prune(self) -> None: + """The post-apply comparison reads ownership as the check does, so it accepts what apply kept.""" + with tempfile.TemporaryDirectory() as temp: + # Resolved as `main` resolves its target, since a Windows temp path can be an 8.3 short name that `run` resolves to its long form. + root = pathlib.Path(temp).resolve() + hub = root / "hub" + (hub / "src").mkdir(parents=True) + (hub / "src" / "carried.txt").write_text("hub", encoding="utf-8") + (hub / "registry").mkdir() + (hub / "registry" / "repos.json").write_text("{}", encoding="utf-8") + (hub / "spec").mkdir() + tree = { + "source": "src", + "target": "dst", + "fidelity": "verbatim-tree", + "appliesTo": "*", + "include": ["**/*"], + "prune": False, + } + (hub / "spec" / "files.json").write_text( + json.dumps({"trees": [tree]}), encoding="utf-8" + ) + target = root / "repo" + (target / "dst").mkdir(parents=True) + (target / "dst" / "own.txt").write_text("mine", encoding="utf-8") + with ( + mock.patch.object(carry, "verify_hub", return_value="0" * 40), + mock.patch.object(carry, "resolve_repo", return_value={"types": ["docs"]}), + mock.patch.object(carry, "verify_target"), + contextlib.redirect_stdout(io.StringIO()), + ): + self.assertEqual(carry.run("apply", "Demo", target, hub), 0) + self.assertEqual((target / "dst" / "carried.txt").read_text(encoding="utf-8"), "hub") + self.assertEqual((target / "dst" / "own.txt").read_text(encoding="utf-8"), "mine") + + def test_a_pruned_declaration_owns_everything_under_its_target(self) -> None: + self.assertEqual(carry.target_patterns({"prune": True, "include": ["*.txt"]}), ["**/*"]) + self.assertEqual(carry.target_patterns({"prune": False, "include": ["*.txt"]}), ["*.txt"]) + + def test_only_a_pruned_declaration_reports_extras(self) -> None: + source = carry.Inventory({"kept.txt": b"hub"}, frozenset(), "source") + target = carry.Inventory( + {"kept.txt": b"hub", "own.txt": b"repo"}, frozenset({"own-dir"}), "target" + ) + pruned = carry.compare_declaration({"prune": True}, source, target) + self.assertEqual((pruned["extra"], pruned["extraDirectories"]), (["own.txt"], ["own-dir"])) + kept = carry.compare_declaration({"prune": False}, source, target) + self.assertEqual((kept["extra"], kept["extraDirectories"]), ([], [])) + def test_empty_source_requires_target_root(self) -> None: with tempfile.TemporaryDirectory() as temp: root = pathlib.Path(temp) diff --git a/tests/test_cli_usage.py b/tests/test_cli_usage.py index f04f7b400..78301f49f 100755 --- a/tests/test_cli_usage.py +++ b/tests/test_cli_usage.py @@ -24,8 +24,8 @@ tree check covers ignored paths and a directory created or emptied anywhere in the tree. The set is discovered from the tree rather than listed, so a new script is held to the rule the -moment it is tracked. An extensionless file opening on a shebang counts. A hook its own host runs -is not a command-line entry point and is listed in NOT_COMMAND_LINE. +moment it is tracked. An extensionless file opening on a shebang counts. A hook its own host runs, +and a library a script sources, is not a command-line entry point and is listed in NOT_COMMAND_LINE. A process the run leaves behind is found by a token in the environment it inherited. That token is read from /proc, so the check runs only on a host that has /proc. On Windows only the PowerShell @@ -69,6 +69,7 @@ "host-setup/agent-safety/claude/stray-process-sweep.py": "a Claude Code hook", "host-setup/agent-safety/claude/tool-containment.py": "a Claude Code hook", "host-setup/agent-safety/claude/test_install.py": "a test module", + "lib/sh/secrets.sh": "a library a script sources", } BEFORE_FLAG = { diff --git a/tests/test_repo_gate.py b/tests/test_repo_gate.py index 451dce762..0c133e7ae 100755 --- a/tests/test_repo_gate.py +++ b/tests/test_repo_gate.py @@ -14,6 +14,7 @@ import functools import importlib import io +import json import os import re import shutil @@ -27,6 +28,7 @@ sys.path.insert(0, str(Path(__file__).resolve().parents[1] / ".github/actions/repo-gate")) import repo_gate +from host_capability import requires_python_3_12, requires_symlink from runner_commands import FORGING_LINES, FORGING_NAMES, ORDINARY_LINES, forged_lines REPO = Path(__file__).resolve().parent.parent @@ -1337,6 +1339,308 @@ def test_a_schema_violation_fails(self) -> None: self.assertTrue(any(h.startswith("schema:") for h in hits)) +class TestCarriedTrees(TreeCase): + """A constructed hub beside the gate and a constructed copy, so each case proves one difference.""" + + def setUp(self) -> None: + super().setUp() + self.hub = self.tmp / "hub" + self.repo = self.tmp / "repo" + self.manifest([self.tree()]) + self.write(self.hub / "lib/sh/secrets.sh", "hub\n") + (self.hub / "scripts").mkdir() + shutil.copy(REPO / "scripts" / "carry.py", self.hub / "scripts" / "carry.py") + self.enterContext(mock.patch.object(repo_gate, "gate_hub_root", return_value=self.hub)) + + @staticmethod + def tree(include: list[str] | None = None, prune: bool = True) -> dict[str, object]: + return { + "source": "lib/sh", + "target": "lib/fleet/sh", + "include": include or ["**/*"], + "prune": prune, + } + + def manifest(self, trees: list[dict[str, object]]) -> None: + self.write(self.hub / "spec/files.json", json.dumps({"trees": trees})) + + def write(self, path: Path, content: str) -> str: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(content, encoding="utf-8") + return path.relative_to(self.repo).as_posix() if path.is_relative_to(self.repo) else "" + + def run_check(self, files: list[str], actions: bool = False) -> tuple[list[str], str]: + with mock.patch.dict(os.environ), contextlib.redirect_stdout(io.StringIO()) as out: + if actions: + os.environ["GITHUB_ACTIONS"] = "true" + else: + os.environ.pop("GITHUB_ACTIONS", None) + hits = repo_gate.check_carried_trees(self.repo, files) + return hits, out.getvalue() + + def test_an_identical_copy_raises_no_difference(self) -> None: + files = [self.write(self.repo / "lib/fleet/sh/secrets.sh", "hub\n")] + self.assertEqual(self.run_check(files), ([], "")) + self.assertEqual( + repo_gate.NOTES, ["compared 1 carried tree(s) against the hub tree beside this gate."] + ) + + def test_a_changed_byte_is_a_note_and_never_a_finding(self) -> None: + files = [self.write(self.repo / "lib/fleet/sh/secrets.sh", "edited\n")] + hits, _ = self.run_check(files) + self.assertEqual(hits, []) + self.assertIn( + "lib/fleet/sh differs from the hub's lib/sh: 1 changed, 0 missing, 0 extra, " + "first secrets.sh.", + repo_gate.NOTES[0], + ) + + def test_a_missing_and_an_extra_file_are_counted(self) -> None: + files = [self.write(self.repo / "lib/fleet/sh/local.sh", "mine\n")] + self.run_check(files) + self.assertIn("0 changed, 1 missing, 1 extra, first secrets.sh.", repo_gate.NOTES[0]) + + def test_a_tree_that_does_not_prune_leaves_a_repository_file_alone(self) -> None: + self.manifest([self.tree(prune=False)]) + files = [ + self.write(self.repo / "lib/fleet/sh/secrets.sh", "hub\n"), + self.write(self.repo / "lib/fleet/sh/local.sh", "mine\n"), + ] + self.run_check(files) + self.assertEqual( + repo_gate.NOTES, ["compared 1 carried tree(s) against the hub tree beside this gate."] + ) + + def test_only_files_a_tree_that_does_not_prune_owns_make_it_carried(self) -> None: + self.manifest([self.tree(include=["fleet_*.sh"], prune=False)]) + self.write(self.hub / "lib/sh/fleet_secrets.sh", "hub\n") + self.run_check([self.write(self.repo / "lib/fleet/sh/own.sh", "mine\n")]) + self.assertEqual( + repo_gate.NOTES, ["compared 0 carried tree(s) against the hub tree beside this gate."] + ) + + def test_an_untracked_file_is_not_a_difference(self) -> None: + files = [self.write(self.repo / "lib/fleet/sh/secrets.sh", "hub\n")] + self.write(self.repo / "lib/fleet/sh/scratch.sh", "untracked\n") + self.run_check(files) + self.assertEqual( + repo_gate.NOTES, ["compared 1 carried tree(s) against the hub tree beside this gate."] + ) + + @requires_symlink + def test_a_link_the_tree_does_not_own_is_refused_as_well(self) -> None: + self.manifest([self.tree(include=["*.sh"], prune=False)]) + files = [self.write(self.repo / "lib/fleet/sh/secrets.sh", "hub\n")] + (self.repo / "lib/fleet/sh/notes.txt").symlink_to(self.hub / "lib/sh/secrets.sh") + _, printed = self.run_check([*files, "lib/fleet/sh/notes.txt"], actions=True) + self.assertEqual( + printed, + "::warning title=carried-trees::lib/fleet/sh holds a link, which a carry refuses\n", + ) + self.assertTrue( + repo_gate.NOTES[0].startswith("lib/fleet/sh could not be compared, since notes.txt") + ) + + @requires_symlink + def test_a_link_in_the_copy_is_refused_as_a_carry_refuses_it(self) -> None: + for prune in (True, False): + with self.subTest(prune=prune): + repo_gate.NOTES.clear() + self.manifest([self.tree(prune=prune)]) + (self.repo / "lib/fleet/sh").mkdir(parents=True, exist_ok=True) + link = self.repo / "lib/fleet/sh/secrets.sh" + link.unlink(missing_ok=True) + link.symlink_to(self.hub / "lib/sh/secrets.sh") + self.assertEqual(self.run_check(["lib/fleet/sh/secrets.sh"]), ([], "")) + self.assertEqual( + repo_gate.NOTES, + [ + ( + "lib/fleet/sh could not be compared, since secrets.sh is a link, " + "which a carry refuses." + ), + "compared 0 carried tree(s) against the hub tree beside this gate.", + ], + ) + + def test_a_tracked_file_deleted_from_the_working_tree_is_missing(self) -> None: + (self.repo / "lib/fleet/sh").mkdir(parents=True) + self.run_check(["lib/fleet/sh/secrets.sh"]) + self.assertIn("0 changed, 1 missing, 0 extra, first secrets.sh.", repo_gate.NOTES[0]) + + def test_a_target_tracked_as_a_link_is_named_rather_than_skipped(self) -> None: + warning = ( + "::warning title=carried-trees::lib/fleet/sh sits at or below a tracked link or file, " + "which a carry refuses\n" + ) + for tracked in ("lib/fleet/sh", "lib/fleet"): + with self.subTest(tracked=tracked): + repo_gate.NOTES.clear() + self.assertEqual(self.run_check([tracked], actions=True), ([], warning)) + self.assertEqual( + repo_gate.NOTES[0], + f"lib/fleet/sh was not compared, since {tracked} is tracked as a link or a file.", + ) + + def test_a_top_level_target_tracked_as_a_link_still_warns(self) -> None: + self.manifest([{**self.tree(), "target": "tools"}]) + printed = "::warning title=carried-trees::tools sits at or below a tracked link or file, which a carry refuses\n" + self.assertEqual(self.run_check(["tools"], actions=True), ([], printed)) + + def test_a_generic_top_level_path_tracked_as_a_file_is_passed_over(self) -> None: + self.assertEqual(self.run_check(["lib"], actions=True), ([], "")) + self.assertEqual( + repo_gate.NOTES, ["compared 0 carried tree(s) against the hub tree beside this gate."] + ) + + def test_a_tree_the_repository_does_not_carry_is_not_compared(self) -> None: + self.run_check([self.write(self.repo / "README.md", "")]) + self.assertEqual( + repo_gate.NOTES, ["compared 0 carried tree(s) against the hub tree beside this gate."] + ) + + def narrowed(self, prune: bool) -> None: + self.manifest([self.tree(include=["*.sh"], prune=prune)]) + self.write(self.hub / "lib/sh/README.txt", "hub only\n") + files = [ + self.write(self.repo / "lib/fleet/sh/secrets.sh", "hub\n"), + self.write(self.repo / "lib/fleet/sh/notes.txt", "repository only\n"), + ] + self.run_check(files) + + def test_a_narrowed_include_narrows_the_hub_side_only_of_a_pruned_tree(self) -> None: + self.narrowed(prune=True) + self.assertIn("0 changed, 0 missing, 1 extra, first notes.txt.", repo_gate.NOTES[0]) + + def test_a_narrowed_include_narrows_both_sides_of_a_tree_that_does_not_prune(self) -> None: + self.narrowed(prune=False) + self.assertEqual( + repo_gate.NOTES, ["compared 1 carried tree(s) against the hub tree beside this gate."] + ) + + @requires_symlink + def test_a_tree_that_cannot_be_compared_is_a_note_and_never_a_finding(self) -> None: + (self.hub / "lib/sh/link").symlink_to(self.hub / "lib/sh/secrets.sh") + files = [self.write(self.repo / "lib/fleet/sh/secrets.sh", "hub\n")] + self.assertEqual(self.run_check(files), ([], "")) + self.assertTrue(repo_gate.NOTES[0].startswith("lib/fleet/sh could not be compared: ")) + self.assertEqual( + repo_gate.NOTES[1], "compared 0 carried tree(s) against the hub tree beside this gate." + ) + + def test_a_carry_that_will_not_load_is_a_note_and_never_a_finding(self) -> None: + (self.hub / "scripts/carry.py").write_text("def broken(:\n", encoding="utf-8") + files = [self.write(self.repo / "lib/fleet/sh/secrets.sh", "hub\n")] + self.assertEqual(self.run_check(files), ([], "")) + self.assertTrue( + repo_gate.NOTES[0].startswith("the hub's scripts/carry.py could not be loaded") + ) + + @requires_python_3_12 + def test_the_hub_carry_runs_under_the_runners_python_3_12(self) -> None: + """A CI runner's system python3 is 3.12, and hub CI otherwise loads carry.py under 3.13 alone.""" + repo = self.tmp / "scanned" + self.write(repo / "lib/fleet/sh/secrets.sh", "stale\n") + subprocess.run(["git", "init", "-q", str(repo)], check=True, timeout=60) + subprocess.run(["git", "-C", str(repo), "add", "-A"], check=True, timeout=60) + gate = REPO / ".github/actions/repo-gate/repo_gate.py" + result = subprocess.run( + [ + "uv", + "run", + "--no-project", + "--no-python-downloads", + "--python", + "3.12", + "python", + "-I", + str(gate), + "--root", + str(repo), + "--check", + "carried-trees", + ], + capture_output=True, + encoding="utf-8", + timeout=300, + check=False, + ) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertIn("lib/fleet/sh differs from the hub's lib/sh: 1 changed", result.stdout) + + def test_a_carry_missing_a_function_the_gate_calls_is_a_note(self) -> None: + carry_file = self.hub / "scripts/carry.py" + text = carry_file.read_text(encoding="utf-8") + carry_file.write_text( + text.replace("def compare_declaration(", "def renamed("), encoding="utf-8" + ) + files = [self.write(self.repo / "lib/fleet/sh/secrets.sh", "hub\n")] + self.assertEqual(self.run_check(files), ([], "")) + self.assertTrue(repo_gate.NOTES[0].startswith("lib/fleet/sh could not be compared: ")) + + def test_an_unreadable_manifest_is_a_note_and_never_a_finding(self) -> None: + self.write(self.hub / "spec/files.json", "{") + self.assertEqual(self.run_check([]), ([], "")) + self.assertTrue(repo_gate.NOTES[0].startswith("no carried tree was compared: ")) + + def test_actions_shows_the_difference_on_the_pull_request(self) -> None: + files = [self.write(self.repo / "lib/fleet/sh/secrets.sh", "edited\n")] + _, printed = self.run_check(files, actions=True) + self.assertEqual( + printed, "::warning title=carried-trees::lib/fleet/sh differs from the hub's lib/sh\n" + ) + + def test_the_annotation_carries_no_name_from_the_scanned_repository(self) -> None: + # Named in the tracked list alone, since a Windows file name cannot hold `:`. + # The hub's own file then reads as missing. + files = ["lib/fleet/sh/x%0A::error::forged"] + _, printed = self.run_check(files, actions=True) + self.assertEqual(printed.count("\n"), 1) + self.assertNotIn("forged", printed) + + def test_outside_actions_nothing_is_printed(self) -> None: + files = [self.write(self.repo / "lib/fleet/sh/secrets.sh", "edited\n")] + self.assertEqual(self.run_check(files)[1], "") + + def test_a_hub_on_another_branch_is_not_read_as_a_stale_copy(self) -> None: + files = ["spec/files.json", self.write(self.repo / "lib/fleet/sh/secrets.sh", "edited\n")] + self.assertEqual(self.run_check(files), ([], "")) + self.assertEqual( + repo_gate.NOTES, + ["the scanned repository is a hub, whose trees are sources rather than copies."], + ) + + +class TestGateHubRoot(TreeCase): + def place(self, relative: str) -> str: + path = self.tmp / relative + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text("", encoding="utf-8") + return str(path) + + def test_the_gate_in_its_hub_tree_finds_the_hub(self) -> None: + self.place("spec/files.json") + gate = self.place(".github/actions/repo-gate/repo_gate.py") + with mock.patch.object(repo_gate, "__file__", gate): + self.assertEqual(repo_gate.gate_hub_root(), self.tmp.resolve()) + + def test_a_lifted_gate_finds_none(self) -> None: + self.place("spec/files.json") + gate = self.place("fetched/repo_gate.py") + with mock.patch.object(repo_gate, "__file__", gate): + self.assertIsNone(repo_gate.gate_hub_root()) + + def test_the_layout_without_a_manifest_finds_none(self) -> None: + gate = self.place(".github/actions/repo-gate/repo_gate.py") + with mock.patch.object(repo_gate, "__file__", gate): + self.assertIsNone(repo_gate.gate_hub_root()) + + def test_a_lifted_gate_reports_that_it_compared_nothing(self) -> None: + with mock.patch.object(repo_gate, "gate_hub_root", return_value=None): + self.assertEqual(repo_gate.check_carried_trees(self.tmp, []), []) + self.assertIn("no hub tree is beside this gate", repo_gate.NOTES[0]) + + class TestExcludeGlobs(unittest.TestCase): """A caller vendoring a subtree it does not author needs `tracked()` narrowed, not re-derived. diff --git a/tests/test_secrets_sh.py b/tests/test_secrets_sh.py new file mode 100755 index 000000000..3b1eb2fc1 --- /dev/null +++ b/tests/test_secrets_sh.py @@ -0,0 +1,322 @@ +#!/usr/bin/env python3 +"""Drive lib/sh/secrets.sh's verdicts and env read against constructed files. + +Each case builds its own tree under a temp root and asks the sourced library about it through a +child bash, so a case proves the function's answer for that shape rather than the state of this +host. A consumer carries the library byte-for-byte, so these cases are the only place it is tested. + +Run as `python3 tests/test_secrets_sh.py`, or under `python3 -m unittest discover -s tests`. +""" + +from __future__ import annotations + +import os +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path + +ROOT = Path(__file__).resolve().parent.parent +LIBRARY = ROOT / "lib" / "sh" / "secrets.sh" +TIMEOUT = 30 +ROOT_ACCOUNT = hasattr(os, "geteuid") and os.geteuid() == 0 + + +def run( + script: str, *args: str, env: dict[str, str] | None = None +) -> subprocess.CompletedProcess[str]: + """Source the library under the fleet header and run `script`, with `args` as `$1`...""" + return subprocess.run( + ["bash", "-c", f'set -Eeuo pipefail; . "$0"; {script}', str(LIBRARY), *args], + capture_output=True, + encoding="utf-8", + timeout=TIMEOUT, + env=env, + check=False, + ) + + +def answer(function: str, *args: str) -> str: + result = run(f'{function} "$@"', *args) + if result.returncode != 0: + raise AssertionError(f"{function} exited {result.returncode}: {result.stderr}") + return result.stdout.rstrip("\n") + + +def write(path: Path, content: str, mode: int = 0o600) -> Path: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_bytes(content.encode()) + path.chmod(mode) + return path + + +@unittest.skipUnless(sys.platform == "linux", "the library runs under bash on a Linux host") +class Tree(unittest.TestCase): + def setUp(self) -> None: + self.temp = tempfile.TemporaryDirectory() + self.root = Path(self.temp.name) + # A temp root can itself sit below a symlink, such as /tmp on some hosts, and the walk reports a component by the spelling it was given. + self.root = self.root.resolve() + + def tearDown(self) -> None: + for current, dirnames, _ in os.walk(self.root): + for name in dirnames: + path = Path(current) / name + if not path.is_symlink(): + path.chmod(0o700) + self.temp.cleanup() + + +class SecretState(Tree): + def test_absent_where_nothing_is_there(self) -> None: + self.assertEqual(answer("secret_state", str(self.root / "missing")), "absent") + + def test_absent_below_a_missing_directory(self) -> None: + self.assertEqual(answer("secret_state", str(self.root / "gone" / "key")), "absent") + + def test_present_for_a_regular_file(self) -> None: + key = write(self.root / "key", "value\n") + self.assertEqual(answer("secret_state", str(key)), "present") + + def test_doubled_and_trailing_slashes_name_the_same_file(self) -> None: + write(self.root / "dir" / "key", "value\n") + self.assertEqual(answer("secret_state", f"{self.root}//dir///key"), "present") + + def test_a_relative_path_is_unknown(self) -> None: + self.assertEqual( + answer("secret_state", "relative/key"), "unknown relative/key is not an absolute path" + ) + + def test_the_root_is_not_a_regular_file(self) -> None: + self.assertEqual(answer("secret_state", "///"), "damaged / is not a regular file") + + def test_a_directory_leaf_is_damaged(self) -> None: + (self.root / "dir").mkdir() + self.assertEqual( + answer("secret_state", str(self.root / "dir")), + f"damaged {self.root}/dir is not a regular file", + ) + + def test_a_file_where_a_directory_belongs_is_damaged(self) -> None: + write(self.root / "file", "") + self.assertEqual( + answer("secret_state", str(self.root / "file" / "key")), + f"damaged {self.root}/file is not a directory, so it can hold no secret", + ) + + def test_a_dangling_leaf_link_is_damaged_rather_than_absent(self) -> None: + (self.root / "key").symlink_to(self.root / "nowhere") + self.assertTrue( + answer("secret_state", str(self.root / "key")).startswith( + f"damaged {self.root}/key is a symlink that does not resolve, so its target is missing" + ) + ) + + def test_a_dangling_parent_link_is_named(self) -> None: + (self.root / "dir").symlink_to(self.root / "nowhere") + self.assertTrue( + answer("secret_state", str(self.root / "dir" / "key")).startswith( + f"damaged {self.root}/dir is a symlink that does not resolve" + ) + ) + + def test_a_fifo_is_not_a_regular_file(self) -> None: + os.mkfifo(self.root / "key") + self.assertEqual( + answer("secret_state", str(self.root / "key")), + f"damaged {self.root}/key is not a regular file", + ) + + @unittest.skipIf(ROOT_ACCOUNT, "root enters a directory whatever its mode") + def test_an_unenterable_directory_is_unknown_rather_than_absent(self) -> None: + locked = self.root / "locked" + locked.mkdir() + locked.chmod(0o600) + self.assertEqual( + answer("secret_state", str(locked / "key")), + f"unknown {locked} cannot be entered by this account", + ) + + @unittest.skipIf(ROOT_ACCOUNT, "the root wording drops the unenterable-directory clause") + def test_the_link_reason_names_an_unenterable_directory_for_a_non_root_account(self) -> None: + self.assertTrue( + answer("unresolved_link_reason").endswith( + " sits behind a directory this account cannot enter" + ) + ) + + +class SecretVerdict(Tree): + def test_a_private_clean_file_is_readable(self) -> None: + key = write(self.root / "key", "value\n") + self.assertEqual(answer("secret_verdict", str(key)), "readable") + + def test_absent_passes_through(self) -> None: + self.assertEqual(answer("secret_verdict", str(self.root / "key")), "absent") + + def test_group_access_is_refused_by_default(self) -> None: + key = write(self.root / "key", "value\n", 0o640) + self.assertEqual( + answer("secret_verdict", str(key)), + f"refused {key} is mode 640, and must give group and other no access", + ) + + def test_shared_allows_group_and_other_read(self) -> None: + key = write(self.root / "key", "value\n", 0o644) + self.assertEqual(answer("secret_verdict", str(key), "shared"), "readable") + + def test_a_link_to_a_private_file_is_refused(self) -> None: + target = write(self.root / "real", "value\n") + (self.root / "key").symlink_to(target) + self.assertEqual( + answer("secret_verdict", str(self.root / "key")), + f"refused {self.root}/key is a symlink, and must be the file itself", + ) + + def test_a_trailing_slash_is_refused_rather_than_normalized(self) -> None: + key = write(self.root / "key", "value\n") + self.assertEqual( + answer("secret_verdict", f"{key}/"), + f"refused {key}/ ends in a slash, so it names a directory rather than a file", + ) + + def test_a_carriage_return_is_refused(self) -> None: + key = write(self.root / "key", "NAME=value\r\n") + self.assertEqual( + answer("secret_verdict", str(key)), + f"refused {key} holds a carriage return, most often from CRLF line endings, " + "and must use LF line endings only", + ) + + def test_a_damaged_state_becomes_a_refusal(self) -> None: + (self.root / "dir").mkdir() + self.assertEqual( + answer("secret_verdict", str(self.root / "dir")), + f"refused {self.root}/dir is not a regular file", + ) + + @unittest.skipIf(ROOT_ACCOUNT, "root reads a file whatever its mode") + def test_an_unreadable_file_is_refused_with_its_owner(self) -> None: + key = write(self.root / "key", "value\n", 0o000) + self.assertTrue( + answer("secret_verdict", str(key)).startswith(f"refused {key} is mode 0 and owned by ") + ) + + def test_a_fifo_content_check_is_unchecked(self) -> None: + os.mkfifo(self.root / "fifo") + self.assertEqual(answer("secret_content_state", str(self.root / "fifo")), "unchecked") + + +class SecretSetVerdict(Tree): + def test_an_empty_set_is_readable(self) -> None: + self.assertEqual(answer("secret_set_verdict"), "readable") + + def test_every_readable_file_makes_the_set_readable(self) -> None: + one = write(self.root / "one", "a\n") + two = write(self.root / "two", "b\n") + self.assertEqual(answer("secret_set_verdict", str(one), str(two)), "readable") + + def test_a_missing_file_outranks_a_refusal_and_names_every_absence(self) -> None: + refused = write(self.root / "refused", "a\n", 0o644) + self.assertEqual( + answer( + "secret_set_verdict", + str(refused), + str(self.root / "first"), + str(self.root / "second"), + ), + "missing first second", + ) + + def test_the_first_refusal_is_the_one_named(self) -> None: + first = write(self.root / "first", "a\n", 0o644) + second = write(self.root / "second", "b\r\n") + self.assertEqual( + answer("secret_set_verdict", str(first), str(second)), + f"refused {first} is mode 644, and must give group and other no access", + ) + + +class RequireSecret(Tree): + def test_a_readable_secret_succeeds_silently(self) -> None: + key = write(self.root / "key", "value\n") + result = run('require_secret "$1"', str(key)) + self.assertEqual((result.returncode, result.stdout, result.stderr), (0, "", "")) + + def test_an_absent_secret_fails_and_says_so(self) -> None: + result = run('require_secret "$1"', str(self.root / "key")) + self.assertEqual(result.returncode, 1) + self.assertEqual(result.stderr, f"ERROR: {self.root}/key is not there\n") + + def test_a_refused_secret_fails_with_the_reason(self) -> None: + key = write(self.root / "key", "value\n", 0o604) + result = run('require_secret "$1"', str(key)) + self.assertEqual(result.returncode, 1) + self.assertEqual( + result.stderr, f"ERROR: {key} is mode 604, and must give group and other no access\n" + ) + + def test_a_bare_call_stops_a_script_under_errexit(self) -> None: + result = run('require_secret "$1"; echo reached', str(self.root / "key")) + self.assertNotEqual(result.returncode, 0) + self.assertNotIn("reached", result.stdout) + + +class ReadEnvValue(Tree): + def test_reads_the_named_value(self) -> None: + env = write(self.root / "app.env", "APP_TOKEN=one\nAPP_HOST=two\n") + self.assertEqual(answer("read_env_value", str(env), "APP_HOST"), "two") + + def test_an_unset_name_reads_empty(self) -> None: + env = write(self.root / "app.env", "APP_TOKEN=one\n") + self.assertEqual(answer("read_env_value", str(env), "APP_HOST"), "") + + def test_the_caller_environment_does_not_stand_in_for_the_file(self) -> None: + env = write(self.root / "app.env", "APP_TOKEN=one\n") + result = run( + 'read_env_value "$1" APP_HOST', str(env), env={**os.environ, "APP_HOST": "stale"} + ) + self.assertEqual((result.returncode, result.stdout), (0, "")) + + def test_a_failing_line_anywhere_fails_the_read(self) -> None: + env = write(self.root / "app.env", "false\nAPP_HOST=two\n") + result = run('read_env_value "$1" APP_HOST', str(env)) + self.assertNotEqual(result.returncode, 0) + + def test_a_failing_line_fails_inside_an_if_condition(self) -> None: + env = write(self.root / "app.env", "false\nAPP_HOST=two\n") + result = run( + 'if read_env_value "$1" APP_HOST; then echo ok; else echo failed; fi', str(env) + ) + self.assertEqual(result.stdout, "failed\n") + + def test_what_the_file_prints_stays_out_of_the_value(self) -> None: + env = write(self.root / "app.env", "echo noise\nAPP_HOST=two\n") + self.assertEqual(answer("read_env_value", str(env), "APP_HOST"), "two") + + def test_the_file_cannot_redirect_the_read_through_the_positional_parameters(self) -> None: + env = write(self.root / "app.env", "APP_HOST=two\nAPP_OTHER=wrong\nset -- x APP_OTHER\n") + self.assertEqual(answer("read_env_value", str(env), "APP_HOST"), "two") + + def test_nothing_the_file_sets_reaches_the_caller(self) -> None: + env = write(self.root / "app.env", "APP_TOKEN=one\nAPP_HOST=two\n") + result = run( + 'read_env_value "$1" APP_HOST >/dev/null; printf "%s" "${APP_TOKEN-unset}"', + str(env), + ) + self.assertEqual(result.stdout, "unset") + + def test_bash_env_does_not_run_ahead_of_the_file(self) -> None: + # A function shadows the builtin, so a hook that ran in the reading bash would answer for the file. + # The outer bash runs it too, and loads the library with `.` rather than `source` for that reason. + hook = write(self.root / "hook.sh", "source() { APP_HOST=injected; }\n") + env = write(self.root / "app.env", "APP_TOKEN=one\n") + result = run( + 'read_env_value "$1" APP_HOST', str(env), env={**os.environ, "BASH_ENV": str(hook)} + ) + self.assertEqual((result.returncode, result.stdout), (0, "")) + + +if __name__ == "__main__": + unittest.main()