Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .editorconfig
Original file line number Diff line number Diff line change
Expand Up @@ -238,7 +238,7 @@ end_of_line = unset
insert_final_newline = false
trim_trailing_whitespace = false

# Caddy map files are space-separated redirect tables generated by `capture/build-redirects.py`.
# Caddy map files are space-separated redirect tables generated by `capture/build_redirects.py`.
# Trailing whitespace is significant to the parse, and the generator owns the formatting.
[deploy/maps/*.map]
trim_trailing_whitespace = false
Expand Down
12 changes: 6 additions & 6 deletions .github/actions/validate/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,29 +21,29 @@ runs:
# This check runs both directions: undocumented values, and rows describing nothing.
- name: Check environment docs step
shell: bash
run: python3 checks/check-env-docs.py
run: python3 checks/env_docs_gate.py

# Hugo copies static/ byte for byte, so whatever a camera wrote is served with the image.
# The build gate reads pages and the prose gate reads text, so nothing else looks inside a file.
# This reads values rather than tag names, and opens archives, since both hid findings before.
- name: Check media metadata step
shell: bash
run: python3 checks/check-media-metadata.py
run: python3 checks/media_metadata_gate.py

# The carried tree is all well formed, so this feeds the same parsers seeded malformed variants.
- name: Fuzz media parsers step
shell: bash
run: python3 checks/fuzz-media-parsers.py
run: python3 checks/media_parser_fuzz_gate.py

# A redacted image replaced by its original passes every other check, so this compares hashes.
- name: Check media redactions step
shell: bash
run: python3 checks/check-media-redactions.py
run: python3 checks/media_redactions_gate.py

# The prose gate reads characters, so this is what reads a post's text for personal data.
- name: Check text personal data step
shell: bash
run: python3 checks/check-text-pii.py
run: python3 checks/text_pii_gate.py

# The hub's validate job checkout is shallow.
# The mtime restore below needs full history to date each file from its last commit.
Expand All @@ -65,7 +65,7 @@ runs:
args: static

# This is the build and URL-contract gate, not an extra step beside one.
# The script runs `hugo --panicOnWarning` and check-url-parity itself, so a separate build would repeat both.
# The script runs `hugo --panicOnWarning` and url_parity_gate.py itself, so a separate build would repeat both.
# --panicOnWarning is the real gate, and the swap overrides in layouts/ let it stay on.
# Every local asset reference must resolve, and every legacy image URL from the WordPress import must too.
#
Expand Down
4 changes: 2 additions & 2 deletions .secrets/example.env
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
#
# WHAT EACH VALUE MEANS IS IN ENVIRONMENT.md, which is the one place it is described.
# This file states the shape and a placeholder. Add a value here and describe it there,
# or checks/check-env-docs.py fails.
# or checks/env_docs_gate.py fails.
#
# One destination: ~/.secrets/blog.<server>.<environment>.env on a workstation, one file
# per environment. The off-host VPS backup pull, and the environment file it reads, are
Expand Down Expand Up @@ -66,7 +66,7 @@ SITE_EXTRA_AUTH_TOKEN=
# Environment-independent, so these belong in the default file only.

# The provenance capture, holding the exports and the crawl of the old platform.
# Every script under capture/ reads beneath it, and all but build-redirects.py write there
# Every script under capture/ reads beneath it, and all but build_redirects.py write there
# too. That one writes the committed maps under deploy/maps/ in the repository.
CAPTURE_ROOT=/path/to/blog-capture

Expand Down
4 changes: 2 additions & 2 deletions .vscode/tasks.json
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@
{
"label": "Check: URL Parity",
"type": "process",
"command": "checks/check-url-parity.py",
"command": "checks/url_parity_gate.py",
"args": [
"public"
],
Expand All @@ -66,7 +66,7 @@
{
"label": "Check: CSP",
"type": "process",
"command": "checks/check-csp.py",
"command": "checks/csp_gate.py",
"args": [
"public"
],
Expand Down
4 changes: 2 additions & 2 deletions AUDIT.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ One key covers both environments, a deliberate decision recorded in `OPERATIONS.
`ENVIRONMENT.md`'s "The GitHub Environments" table declares every secret and variable name, its kind, and each store holding it. This repo's own check compares that table against GitHub:

```sh
python3 checks/check-github-env.py
python3 checks/github_env_gate.py
```

It compares in both directions and reports three kinds of finding. A store lacks a declared name, a store holds a name the table does not list, or a value is held as the wrong kind. The second is the one presence-checking misses. A Pangolin access token on `production` is reported as unlisted rather than passed as a harmless extra. Production answers unauthenticated, so a token there means a check could pass through a gate production is not supposed to have.
Expand All @@ -76,7 +76,7 @@ This site's whole risk is silent URL loss, so the contract is audited like any o

```sh
hugo --gc --minify --panicOnWarning
checks/check-url-parity.py public
checks/url_parity_gate.py public
checks/check-live-urls.sh <base-url>
```

Expand Down
32 changes: 16 additions & 16 deletions CONTENT.md

Large diffs are not rendered by default.

6 changes: 3 additions & 3 deletions ENVIRONMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

Every configuration value this repository reads or writes, described once. [`OPERATIONS.md`](./OPERATIONS.md) is the procedure and this is the reference the procedure points at, so a value is explained here and named elsewhere.

**A description lives here and nowhere else.** The `.example` files state the format, the scripts state the defaults, and this file states what a value means. [`checks/check-env-docs.py`](./checks/check-env-docs.py) fails if a value is declared or consumed anywhere and has no row below, or has a row and is declared nowhere, so the two stay in step without depending on anyone remembering.
**A description lives here and nowhere else.** The `.example` files state the format, the scripts state the defaults, and this file states what a value means. [`checks/env_docs_gate.py`](./checks/env_docs_gate.py) fails if a value is declared or consumed anywhere and has no row below, or has a row and is declared nowhere, so the two stay in step without depending on anyone remembering.

**Values are grouped by where they live rather than by what reads them**, because where a value lives determines who can change it, what happens when it is wrong, and whether it reaches a public history.

Expand Down Expand Up @@ -30,7 +30,7 @@ Held in `~/.secrets/blog.<server>.<environment>.env`, one file per environment,
| `SITE_EXTRA_BASE_URL` | the family site's base URL in this environment | Read by `check-live-urls.sh`, which checks the family page there when it is set. Leave it unset until the family hostname reaches this environment, since the check fails on any other answer. |
| `SITE_EXTRA_AUTH_TOKEN_ID` | the family resource's access token id, for a family site behind the auth gate | Set both or neither, and only with `SITE_EXTRA_BASE_URL`. A token opens exactly one proxy resource, so the blog's pair cannot open the family host. |
| `SITE_EXTRA_AUTH_TOKEN` | the token itself | Sent as `P-Access-Token` to the family host and nowhere else. |
| `CAPTURE_ROOT` | the provenance capture, holding the WordPress exports, the crawl of the old platform, and the inventories derived from it | Every script under [`capture/`](./capture/) reads beneath it, and all but one write there too. The exception is [`capture/build-redirects.py`](./capture/build-redirects.py), which writes the committed maps under `deploy/maps/` in this repository, and which also accepts the capture as a first argument that wins over this value. Environment-independent, so it belongs in the default file only. |
| `CAPTURE_ROOT` | the provenance capture, holding the WordPress exports, the crawl of the old platform, and the inventories derived from it | Every script under [`capture/`](./capture/) reads beneath it, and all but one write there too. The exception is [`capture/build_redirects.py`](./capture/build_redirects.py), which writes the committed maps under `deploy/maps/` in this repository, and which also accepts the capture as a first argument that wins over this value. Environment-independent, so it belongs in the default file only. |
| `CAPTURE_SOURCE_URL` | the old platform's base URL, the site the crawl and the URL verification ran against | **Not `SITE_BASE_URL`.** The two hold the same string after the cutover and mean different things, so merging them points a verification run at the new site while every check still passes. Environment-independent. |
| `CAPTURE_SOURCE_API` | the old platform's REST API for that site, carrying its numeric site id | Read for the post and page bodies in **rendered** form, which is what expands shortcodes so a media reference is seen the way a reader's browser sees it. Environment-independent. |
| `CAPTURE_AUTHOR_SLUG` | the old platform's author slug, used to backfill the author archive and its pagination | Optional, and an account name rather than a site value, which is why it is a variable at all. Unset, [`capture/classify.py`](./capture/classify.py) skips the backfill and says so, rather than emitting a list that is silently short by the author URLs. Environment-independent. |
Expand Down Expand Up @@ -60,7 +60,7 @@ Every variable and secret this repository stores for GitHub Actions, Dependabot,
| `CODEGEN_APP_PRIVATE_KEY` | secret | `repository`, `dependabot` | the merge bot's GitHub App private key, read the same way |
| `CODECOV_TOKEN` | secret | `repository`, `dependabot` | the Codecov upload token, passed by every workflow that calls the hub validator. Its coverage upload is best-effort, so an empty token skips the upload and fails nothing. The `dependabot` copy is what a Dependabot pull request's run reads |

**[`checks/check-github-env.py`](./checks/check-github-env.py) compares this table against GitHub.** It reads the names those stores hold, never a value, and leaves the Codespaces secret store unread. It reports a value missing from a store, held on a store the table does not name, or held as the wrong kind. Every environment GitHub has is read, so an environment the table never names must hold nothing. Run it after changing a value on GitHub or a row here. It needs a `gh` login that can administer the repository, since listing environment secrets requires that. A workflow's `GITHUB_TOKEN` cannot, so CI does not run it.
**[`checks/github_env_gate.py`](./checks/github_env_gate.py) compares this table against GitHub.** It reads the names those stores hold, never a value, and leaves the Codespaces secret store unread. It reports a value missing from a store, held on a store the table does not name, or held as the wrong kind. Every environment GitHub has is read, so an environment the table never names must hold nothing. Run it after changing a value on GitHub or a row here. It needs a `gh` login that can administer the repository, since listing environment secrets requires that. A workflow's `GITHUB_TOKEN` cannot, so CI does not run it.

**The family values take the hub's names for a second site.** The hub's deploy task reads `SITE_EXTRA_BASE_URL` and the `SITE_EXTRA_AUTH_TOKEN_ID`/`SITE_EXTRA_AUTH_TOKEN` pair by those names from the environment. `check-live-urls.sh` and the `~/.secrets/blog.*.env` files use the same three names, so nothing translates between them. A deploy therefore checks the family host with its own token.

Expand Down
2 changes: 1 addition & 1 deletion GOVERNANCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -270,7 +270,7 @@ Every repo's GitHub repository details (the About panel) follow a fixed conventi

- [`AGENTS.md`](./AGENTS.md): the agent entry point, carrying context and delegation rules plus the map to the sections below.
- [`GOVERNANCE.md`](./GOVERNANCE.md), [`CODESTYLE.md`](./CODESTYLE.md), [`WORKFLOW.md`](./WORKFLOW.md), [`OPERATIONS.md`](./OPERATIONS.md), [`AUDIT.md`](./AUDIT.md): the governance, operations, and audit docs. This file is the cross-cutting-rules authority.
- [`content/`](./content/): the posts and pages, one file per URL. An imported archive that keeps its own voice. Its only text changes are what `scripts/normalize-text.py` applies: ASCII character substitutions, and a manifest of doubled-word and grammar corrections, a title's HTML entity, and two links that located the home. A blockquote, a fenced code block, and front matter keep the characters they were written with. [`CONTENT.md`](./CONTENT.md) is the contract for writing a new one.
- [`content/`](./content/): the posts and pages, one file per URL. An imported archive that keeps its own voice. Its only text changes are what `scripts/normalize_text.py` applies: ASCII character substitutions, and a manifest of doubled-word and grammar corrections, a title's HTML entity, and two links that located the home. A blockquote, a fenced code block, and front matter keep the characters they were written with. [`CONTENT.md`](./CONTENT.md) is the contract for writing a new one.
- [`static/`](./static/): the media the posts reference, verified by content hash against the source export.
- [`layouts/`](./layouts/), [`themes/`](./themes/), [`assets/`](./assets/): the site's templates and shortcodes, the vendored theme, and the site styles added to its stylesheet.
- [`checks/`](./checks/): the URL contract and the gates that enforce it, covering both the built output and a running server. Its `pyproject.toml` holds the ruff, mypy, and coverage config for these standard-library-only gates.
Expand Down
10 changes: 5 additions & 5 deletions OPERATIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,17 +94,17 @@ Expect a `PASS` naming the number of URLs honored. That is the three lists' comb

A documentation-only or workflow-only change does not need this. A change to the four paths above does, because for those CI's green is not evidence.

**A change to a `Content-Security-Policy` in `security-headers.txt` is also tried in a browser**, on the local staging mirror, before its pull request opens. So is a theme or `layouts/` change that adds a style, a font, a fetch, or a script. [`checks/check-csp-browser.py`](./checks/check-csp-browser.py) loads every page of `golden-urls.txt` plus `/search/` in headless Chromium. It scrolls each page to its end, clicks its code-copy buttons, and types a search query. It reads each `securitypolicyviolation` event from the page and from each frame in it. It reports them by directive and blocked origin, with the first page each occurred on. The event fires for an enforcing policy and a `Content-Security-Policy-Report-Only` one alike, so a trial can run Report-Only first:
**A change to a `Content-Security-Policy` in `security-headers.txt` is also tried in a browser**, on the local staging mirror, before its pull request opens. So is a theme or `layouts/` change that adds a style, a font, a fetch, or a script. [`checks/csp_browser_gate.py`](./checks/csp_browser_gate.py) loads every page of `golden-urls.txt` plus `/search/` in headless Chromium. It scrolls each page to its end, clicks its code-copy buttons, and types a search query. It reads each `securitypolicyviolation` event from the page and from each frame in it. It reports them by directive and blocked origin, with the first page each occurred on. The event fires for an enforcing policy and a `Content-Security-Policy-Report-Only` one alike, so a trial can run Report-Only first:

```sh
set -e
set -a; . ~/.secrets/blog.local.staging.env; set +a
checks/check-csp-browser.py "$SITE_BASE_URL"
checks/csp_browser_gate.py "$SITE_BASE_URL"
```

- **It needs Docker and `uv` on the host, and no browser.** Chromium runs in the Playwright image [`checks/browser/Dockerfile`](./checks/browser/Dockerfile) pins, where Dependabot moves it. `uv` installs the Playwright package inside it at the image tag's own version, so the two cannot drift. Its cache is `UV_CACHE_DIR` where that is set, or a directory under the system temporary root.
- **The container shares the host network**, so it reaches a local mirror by the name the host resolves.
- **A violation fails the run unless [`checks/csp-browser-allow.txt`](./checks/csp-browser-allow.txt) lists it.** Each line is a directive, a blocked origin, and the reason, tab separated. A blocked scheme is written as the browser reports it, such as `data` with no colon. An entry that nothing matched fails a run over the default list, so an entry goes when its cause does. `--urls` takes another list, and such a partial run leaves unused entries alone.
- **A violation fails the run unless [`checks/csp-browser-allow.txt`](./checks/csp-browser-allow.txt) lists it.** Each line is a directive, a blocked origin, and the reason, tab separated. A blocked scheme is written as the browser reports it, such as `data` with no colon. An entry that nothing matched fails a run over the default list, so an entry goes when its cause does. `--urls` takes another list, and such a partial run leaves unused entries alone. It also loads `/search/` only where the list names it, since a list can be for the family site, which has no search page.
- **A page that does not answer `200` fails the run**, since a trial over error pages proves nothing.
- **It runs only against the local mirrors.** VPS staging sits behind the auth gate, which the check does not present. A full run is hundreds of pages. Production takes only a handful of probes per pass, since a burst there can get the reviewing network banned.
- **It runs by hand rather than in the release.** It needs Docker, a browser image, and a running server, and neither the release build nor the unit suite has all three.
Expand Down Expand Up @@ -214,7 +214,7 @@ ssh "$VPS_SSH_HOST" true && echo reachable

**The off-host copy, maintained outside this repository, writes the rotated access logs to `LOG_ARCHIVE_ROOT`, and this section covers nothing more about it.** Its installation, how the VPS itself is provisioned, and its trust model are the backup host's own configuration to document, not this repository's. What "Logs and Debugging" needs from its schedule and copy behavior, to read the logs correctly, is covered there instead. Read the unit and its last run on the backup host rather than trusting a schedule written down anywhere, including here.

**`LOG_ARCHIVE_ROOT` is spelled the same way on both sides, so there is nothing to reconcile.** The pull writes it and the log review reads it, under the one name. Every value this repository reads or writes is described once, in [`ENVIRONMENT.md`](./ENVIRONMENT.md), and [`checks/check-env-docs.py`](./checks/check-env-docs.py) fails if one is declared without a description or described without existing.
**`LOG_ARCHIVE_ROOT` is spelled the same way on both sides, so there is nothing to reconcile.** The pull writes it and the log review reads it, under the one name. Every value this repository reads or writes is described once, in [`ENVIRONMENT.md`](./ENVIRONMENT.md), and [`checks/env_docs_gate.py`](./checks/env_docs_gate.py) fails if one is declared without a description or described without existing.

```sh
set -a; . ~/.secrets/blog.local.production.env; set +a
Expand Down Expand Up @@ -604,7 +604,7 @@ Ordering is load-bearing, so every redirect lives in a single `route` block. Out
| Term archives the generator does not build | `terms.map` |
| A `slugs.map` or `terms.map` key without its trailing slash, on a path with no dot | A second lookup of both maps, keyed on the path with the slash added |

The maps are generated by [`capture/build-redirects.py`](./capture/build-redirects.py) and the generated files are committed, so a deploy never regenerates them. How it selects its input, and why that selection is the part to get right, is in [`capture/README.md`](./capture/README.md).
The maps are generated by [`capture/build_redirects.py`](./capture/build_redirects.py) and the generated files are committed, so a deploy never regenerates them. How it selects its input, and why that selection is the part to get right, is in [`capture/README.md`](./capture/README.md).

### The Family Site

Expand Down
Loading
Loading