feat: record durable run history in a ClickHouse table

[lesandie]

Stacked on #6. Base is test/pin-deletion-scope, so this PR's diff shows only its own changes and there are no conflicts. GitHub retargets it to master automatically once #6 merges. Review #6 first.

Why

Pod logs are not a record. The kubelet rotates container output (10Mi over 5 files by default), so kubectl logs cannot return the beginning of a long run, and ttlSecondsAfterFinished deletes the Job and its pods along with everything they printed. A cleanup that reclaimed terabytes left no evidence of what it did once that window closed.

What

Each run appends structured events to <COLLECTTABLEPREFIX><disk>_log, beside the auxiliary table and never truncated.

Events	phase start · throttled collect progress · per-sample start · per-delete-batch checkpoint · finish with attempt totals · warnings · errors
Columns	event_time, run_id, phase, event, message, objects, bytes, + scope: bucket, prefix, disk, cluster, dry-run, ClickHouse host, hostname
Identity	S3GC_RUNID defaults to a generated timestamped id; the Job template passes JOB_NAME
Opt out	--runlog false / S3GC_RUNLOG_FLAG=false / RUNLOG=false

Rows are self-describing evidence rather than lines of text, so the questions an operator actually asks are one query:

-- how far did a failed delete get, before deciding to start a replacement Job?
SELECT max(objects) AS deleted, max(bytes) AS reclaimed
FROM   <db>.<prefix><disk>_log
WHERE  run_id = '<job-name>' AND event = 'checkpoint';

Why ClickHouse, and not a volume or the bucket

• ClickHouse — the connection, credentials and grants already exist. No new infrastructure, no new permission.
• Not a volume — the container is readOnlyRootFilesystem: true; an emptyDir dies with the pod, and a PVC means provisioning, RWO scheduling and cleanup for a one-shot Job.
• Not the bucket — it needs a new s3:PutObject grant, and an object written under S3PATH would be listed by the next collect, found absent from system.remote_data_paths, and become a deletion candidate. s3gc would garbage-collect its own logs.

Three constraints, each with a test

This is bookkeeping attached to an irreversible operation, so most of the tests are about what it must not do.

1. Writes go on ch_writer, never ch_client. do_use() holds ch_client's session for the whole anti-join stream, and a second query on a held session is SESSION_IS_LOCKED (373) — the 0.6.0 defect, which would now fire mid-delete. test_run_log_writes_off_the_streaming_session uses the existing StreamingCH fake to prove it.
2. A logging failure never fails the run. One failure disables the run log for the rest of the process rather than retrying every batch of a multi-hour delete; a missing CREATE TABLE grant degrades to stdout only with one warning.
3. Messages are redacted through the existing LogFormatter._filter before insert, so a credential cannot reach a table that outlives the run.

Also fixed

Collect now reports progress at INFO, throttled to every 100 000 objects. Per-batch progress was DEBUG-only, so a multi-hour collect over millions of objects emitted about four lines at --verbose — while --debug emits one line per object, which on a large bucket exceeds the kubelet's rotation limit and destroys the beginning of its own output. The real choice was "almost nothing" or "too much to retrieve".

The deletion scope is untouched

No change to the anti-join or to what gets deleted. Re-ran #6's mutation testing against this branch: 11 of 11 still caught.

Deferred

Two items added to TODO.md rather than fixed here: print() output is block-buffered (no PYTHONUNBUFFERED in the image) so bare prints including the closing s3gc: OK are lost when activeDeadlineSeconds fires — the run log covers the evidence case, but stdout fidelity is a separate change; and a possible TTL on the run-log table, which grows without bound across many cleanups.

Checks

pytest -m "not dev_cluster"      93 passed, 1 xfailed
render.py example.env            OK
kubeconform -strict              Valid: 1, Invalid: 0, Errors: 0
RUNLOG=false                     renders S3GC_RUNLOG_FLAG="false"
RUNLOG=maybe                     render error, exit 64
env file with no RUNLOG          renders, defaults to true
mutation suite (#6's eleven)     11/11 caught

🤖 Generated with Claude Code