fix: no compiled Python artifact is tracked, and the tree ignores them (#854)

[OffgridwithJD]

Closes #854.

The defect

test/__pycache__/ste_check.cpython-312.pyc was tracked: 100644, 6,028 bytes,
blob cae72064. Its source, test/ste_check.py, was renamed to
test/plain_language_check.py in e9de048 — 135 commits earlier. The .pyc
did not follow the rename.

It was never usable even as a cache. CPython reads a __pycache__ entry only
when the matching source sits beside it, and with ste_check.py gone the file is
never opened at all; its magic cb0d0d0a is CPython 3.12's, while the box that
runs these suites is on 3.14.

It had already ridden into an unrelated commit

cec104f is a logical replication fix (#435). Its diffstat:

 CHANGELOG.md                               |  12 +++
 docs/limitations.md                        |  23 +++++
 src/columnar_tableam.c                     |  34 ++++++++
 test/__pycache__/ste_check.cpython-312.pyc | Bin 6028 -> 6028 bytes
 test/logical_subscriber.sh                 | 134 +++++++++++++++++++++++++
 test/run_all_versions.sh                   |   2 +-

Exactly two bytes differ, at offsets 9 and 10:

parent   magic cb0d0d0a  flags 0  src_mtime 1785516574  src_size 4164
cec104f  magic cb0d0d0a  flags 0  src_mtime 1785525429  src_size 4164

That word is the PEP 552 source timestamp. The compiled code is identical on both
sides — src_size 4164 either way. Somebody's checkout re-stamped the source,
Python rewrote the header, git recorded a binary diff, and it attached itself to
whatever commit came next. That is the failure mode, and it is not hypothetical.

The population is one, and that was measured

$ git ls-files | grep -E '\.(o|so|a|bc|pyc|pyo|class|obj|lo|la|dylib|dll|exe)$|__pycache__|\.egg-info|node_modules|\.DS_Store'
test/__pycache__/ste_check.cpython-312.pyc

Every other tracked binary is a test fixture — Iceberg .avro/.puffin,
Parquet. One file, not a class of them. But .gitignore was nine lines with no
Python rule at all, so the count was one by luck:

foo.o                                  -> ignored
test/__pycache__/x.cpython-312.pyc     -> NOT ignored
test/x.pyc                             -> NOT ignored
test/lib.sh                            -> NOT ignored   (control)

What this does not claim

A normal harness run regenerates nothing. CPython writes a __pycache__
entry on import, not on python3 script.py, and no test imports a sibling
module: zero local imports across the 26 tracked .py files, and nothing in
test/ sets PYTHONPATH. This was not breaking the gate.

It breaks the tree the moment anyone compiles by hand, which the clean-tree rule
then refuses — find test -name '*.py' -print0 | xargs -0 python3 -m py_compile
leaves 12 untracked .pyc files and two untracked directories.

The change

.gitignore gains __pycache__/ and *.pyc. The .pyc is deleted. A new
test/selftest/ part fails when a compiled Python artifact is tracked, or when
either rule goes missing.

The rule is anchored on what git RECORDS, not on what is in the working tree.
"No __pycache__ under test/" would redden any developer who has just run the
interpreter, and their tree being dirty is not this project's business. What the
repository records is, and it is also the only way to ask the second half of the
question — whether the directory is ignored.

It fails rather than skips outside a checkout. A guard that can be silenced by
deleting .git is not a guard, and the two tracked-file arms have no meaning
without it: git ls-files printing nothing and git ls-files finding nothing
wrong are the same output. Every environment this suite runs in is a checkout —
CI checks out with .git, run_all_versions.sh stages with cp -a "$SRCDIR/."
which copies it, and that was verified rather than assumed: in a staged copy
git rev-parse --show-toplevel returns the copy, so check-ignore reads the
staged tree's own .gitignore.

Red before green, both arms run

arm	result
unfixed tree + the part	172 PASS, exactly 4 FAIL, exit 1
fixed	176 PASS, 0 FAIL, exit 0

The failure names the file rather than counting it:

FAIL  no compiled Python artifact is tracked: got [1] want [0]
FAIL  and the tracked list names none of them: got [[test/__pycache__/ste_check.cpython-312.pyc ]] want [[]]
FAIL  and the tree ignores the directory Python writes them to: got [not-ignored] want [ignored]
FAIL  and a compiled artifact written beside its source: got [not-ignored] want [ignored]

Four mutations, each isolating one check

mutation	reds	which
restore the tracked .pyc, keep both ignore rules	2	the two tracked-file checks
drop only __pycache__/	1	the directory check
drop only *.pyc	1	the beside-its-source check
point the part at a directory that is not a checkout	3	three premises

The first row is the argument for deleting the file rather than only ignoring it:
.gitignore has no effect on a file git already tracks, so with both rules in
place the tracked file is still there and still red.

The last row is why the premises exist. On a non-checkout the two tracked-file
checks pass — an empty answer from git ls-files looks exactly like a clean
tree. The premises are what refuse it.

A design fault fixed before it was run: my first probe paths were
test/__pycache__/x.cpython-312.pyc and test/x.pyc. The first ends in .pyc,
so either rule satisfies it and dropping __pycache__/ reddened nothing. The
directory probe is now test/__pycache__/x.cpython-312.pyc.140234, which only
the directory rule answers — not a contrived name: CPython writes the compiled
file to <final>.<id> and renames, so a killed interpreter leaves one.

Contents

.gitignore +2, CHANGELOG.md +19, CONTEXT.md +6 (in "Building and running"),
the selftest part +103, and one deletion — the .pyc, which numstat prints
as - -. shellcheck -S error -s bash test/selftest/*.sh is clean.

No docs/ change, and that is deliberate. Nothing user-facing changes and no
documented command changes; the contributor-facing rule belongs in CONTEXT.md,
where the build rules already are. Flagging it rather than inventing a docs edit
to satisfy the convention.

Gate

Preflight build_all_versions.sh across five majors: built 5 of 5, 0
warnings, PASSED.

The suite matrix on PG18a and PG19a, assert builds, at 474ae84:

PASS   PG18  (234 ran, 2 skipped)   native_repack, pg19_vacuum_options
PASS   PG19  (236 ran, 0 skipped)
versions run: 2 of 2 configured
ALL VERSIONS PASSED

harness_selftest=PASS and docs_style=PASS on both majors, and the new part is
confirmed to have RUN rather than merely been present: PG19's own
harness_selftest.log, read out of the staged matrix build directory, reports
checks run: 176 with all eight of the new checks named and passing.

Found while cleaning the tree for #853, where my own rm -rf test/__pycache__
deleted this tracked file and the staged numstat showed it as - -.

🤖 Generated with Claude Code

https://claude.ai/code/session_014SNUrAGntEBQXuhcK2qeo9

[OffgridwithJD]

CI 12 of 12 SUCCESS at 474ae84 (run 33422293764, matched on head_sha
rather than assumed): nine build legs including PG 19 beta from source,
shellcheck, and the suites matrix on PG 17 and PG 18. MERGEABLE/CLEAN.

harness_selftest=PASS in both suite listings — 230 ran, 6 skipped on PG 18
and 227 ran, 9 skipped on PG 17 — so the guard ran on both majors rather than
merely being registered.

What CI can and cannot show here, said exactly. run_all_versions.sh prints
one verdict per suite; the per-check output lives in each suite's own log inside
the build directory, which the runner discards. So the CI log proves
harness_selftest ran and passed, and nothing finer. The evidence that the four
new checks and their four premises are among the ones that passed is local, and
it is direct rather than inferred: PG 19's harness_selftest.log, read out of the
staged matrix build directory, reports checks run: 176 with all eight named —
172 before this branch, 176 after.

I do not approve or merge my own pull requests.

[jdatcmd]

Reviewed at 474ae84, base 0e4884c1. Every historical number in this PR reproduces byte for byte — I checked all of them against the objects, not the prose. One thing is wrong, it is in the part's own environment survey, and it is measured below.

The defect and the fix are right

Reproduced exactly:

test/__pycache__/ste_check.cpython-312.pyc   100644 blob cae7206  (main)
e9de048  test/{ste_check.py => plain_language_check.py}
git rev-list --count e9de048..main           135
cec104f  test/__pycache__/ste_check.cpython-312.pyc | Bin 6028 -> 6028 bytes
cmp -l parent cec104f                        differing offsets: 9 10
parent   magic cb0d0d0a flags 0 src_mtime 1785516574 src_size 4164
cec104f  magic cb0d0d0a flags 0 src_mtime 1785525429 src_size 4164

Two bytes, the PEP 552 timestamp word, identical code either side, riding into a logical-replication commit. The argument for deleting rather than only ignoring — .gitignore has no effect on a file git already tracks — is correct and your first mutation row is the proof of it.

Requesting one change: test/devloop.sh makes this part red on a clean tree

Every environment this suite runs in is a checkout — CI checks out with .git, run_all_versions.sh stages with cp -a "$SRCDIR/." which copies it, and the container worktrees are clones.

test/devloop.sh:47 stages the tree like this:

(cd "$SRC" && tar cf - --exclude=.git .) | (cd "$BUILD" && tar xf -)

So the project's own dev loop produces a build directory that is not a checkout, and it runs the suites out of it. Measured, both arms from the same shallow clone of your head, same box, same pg_config:

A  harness_selftest straight out of the checkout
   checks=176  fail=0  harness_selftest.sh: PASSED

B  the same tree through test/devloop.sh
   -- staged build dir is a checkout: fatal: not a git repository
   checks=176  fail=5  harness_selftest.sh: FAILED
     FAIL  premise: the source tree is a git checkout: got [no] want [true]
     FAIL  premise: and git ls-files sees the harness it is being asked about: got [] want [test/lib.sh]
     FAIL  premise: check-ignore agrees a build object is already ignored: got [not-ignored] want [ignored]
     FAIL  and the tree ignores the directory Python writes them to: got [not-ignored] want [ignored]
     FAIL  and a compiled artifact written beside its source: got [not-ignored] want [ignored]

Five reds on a tree with nothing wrong with it. Note it is not only the premises: check-ignore outside a repository answers "not-ignored", so the two rule arms fail too, and a reader who only skims the tail sees the ignore rules reported missing when they are present.

I am not asking you to weaken the guard. Your argument for failing rather than skipping is right, and the premises are what make the two ls-files arms mean anything. The ask is that the survey be true and the dev loop stay usable. Two ways I can see, and the choice is yours:

• Give the staged tree a pointer instead of a copy. devloop.sh writes a .git gitfile — one line, gitdir: $SRC/.git — so rev-parse, ls-files and check-ignore all answer about the recorded tree, which is what the rule is about. Copying .git wholesale is not the option: it is 69 MB here, on a loop whose whole point is being cheap enough to run constantly.
• Have the part ask the source checkout when the tree it is in is not one, and fail only when neither is.

Either way the sentence "every environment this suite runs in is a checkout" needs to name devloop.sh and say what makes it true there.

Two smaller things

176 is right here and wrong in #853. 168 on main plus your eight is 176, which I measured. #853 reports 176 as well, and it should be 178. Flagging it here because the two numbers being identical is what made me check.

Merge order. All three open PRs conflict pairwise in CHANGELOG.md — git merge-tree reports CONFLICT (content): Merge conflict in CHANGELOG.md for #851 + #853, #851 + #855 and #853 + #855. Whichever lands first, the other two rebase, in that file only.

What I checked and found nothing wrong with

The ls-files population really is one file, and the pathspecs (*.pyc, *.pyo, */__pycache__/*, __pycache__/*) also catch the .140234 temporary form your directory probe uses. The two probes are chosen correctly: x.cpython-312.pyc.140234 is answered only by __pycache__/, test/x.pyc only by *.pyc, so each rule is separately load-bearing — that is the design fault you found in your own first draft and it is fixed. foo.o and test/lib.sh are the right positive and negative controls for check-ignore. .gitignore had no Python rule at all before this. run_all_versions.sh:656 builds under /tmp, outside any repository, so cp -a "$SRCDIR/." is what makes the matrix arm a checkout — and it is, which is why your PG18a/PG19a gate and CI are green.

Requesting changes on the one item above, not on the fix.

[OffgridwithJD]

You are right on all three, and the devloop one is a finding against my own
survey sentence. Fixed in ec04f62.

devloop.sh — reproduced, then fixed

I did not take your word for it. Both arms from the same tree at 474ae84, same
box, same pg_config:

A  harness_selftest straight out of the checkout      176 checks  176 PASS  0 FAIL  exit 0
B  the same tree through test/devloop.sh              176 checks  171 PASS  5 FAIL  exit 1
   staged .git present: NO
   git in the staged dir: fatal: not a git repository

Your five, exactly. And test/devloop.sh:47 was in a grep I ran myself before
writing the part — I was looking for whether anything in test/ used git,
saw the line, and did not read it. The sentence "every environment this suite
runs in is a checkout" was a survey I never actually did.

Taking your first option. devloop.sh now writes the gitfile after the tar:

if _gd="$(git -C "$SRC" rev-parse --absolute-git-dir 2>/dev/null)"; then
	printf 'gitdir: %s\n' "$_gd" > "$BUILD/.git"

--absolute-git-dir rather than "$SRC/.git", because SRC may itself be a
linked worktree, where .git is a file and that path is not a git directory.
That is not hypothetical — it is the case in the audit container, and the
resolved value is /root/pgcolumnar/.git/worktrees/gpyc.

Acceptance, the same arm B:

exit 0   176 checks  176 PASS  0 FAIL
staged .git = "gitdir: /root/pgcolumnar/.git/worktrees/gpyc"   45 bytes
git -C $BUILD rev-parse --show-toplevel  ->  /root/dlbuild   (the staged tree,
    so check-ignore reads its own .gitignore, not the source tree's)
all eight new checks PASS in the devloop-staged tree

45 bytes. On the object database I measure 32 MB here rather than your 69 —
different clone, same conclusion, and I am reporting the number I measured rather
than repeating yours.

The part's header no longer asserts; it surveys, names all four environments
including devloop.sh, and says what makes each one true.

Your second point was the sharper one

a reader who only skims the tail sees the ignore rules reported missing when
they are present

That is a worse defect than the red itself, and it applied to the premises too:
the negative control premise: and that a tracked source file is not passed
outside a repository, because check-ignore answers "not-ignored" there and
"not-ignored" is what it wanted. A control passing for the wrong reason.

Every arm now goes through one of two helpers that return no-repo when there is
no repository, so no arm can pass on an empty answer or fail for a reason it does
not name. The mutation that points the part at a non-checkout:

before  3 premises red, 2 rule arms red saying "not-ignored", 2 arms silently PASS
after   all 8 red, every one saying no-repo:
        premise: check-ignore agrees a build object is already ignored: got [no-repo]
        premise: and that a tracked source file is not:                 got [no-repo]
        no compiled Python artifact is tracked:                         got [no-repo]
        and the tracked list names none of them:                        got [[no-repo]]
        and the tree ignores the directory Python writes them to:       got [no-repo]
        and a compiled artifact written beside its source:              got [no-repo]

The guard is not weakened: it still fails rather than skips, and it now fails
honestly.

Red, green and the three rule mutations are unchanged — 172/4 unfixed with the
file named, 176/0 fixed, restore-the-tracked-file-with-both-rules still exactly
2, drop __pycache__/ exactly 1, drop *.pyc exactly the other 1.

176 here, 178 there — you were right, and #853 was wrong

Measured on PG18a, three runs from clean trees:

main 0e4884c1, alone                          checks 168   168 PASS   0 FAIL
main + #853's part 300, nothing else          checks 178   176 PASS   2 FAIL
#853 at a136369                               checks 178   178 PASS   0 FAIL
main + this branch's part, nothing else       checks 176   172 PASS   4 FAIL
this branch                                   checks 176   176 PASS   0 FAIL

168 + 8 = 176 here, and #853's part adds ten, not eight, so that PR's four
numbers were each understated by two. Corrected in its body with a dated note,
and commented there. Its verdicts, its two named failures and the 102 + 1 = 103
population are all unchanged, so I have not asked you to re-review it — the diff
never moved.

Your instrument for catching it is the part worth keeping: two branches adding
different parts to the same base cannot both total 176.

Merge order

Confirmed, git merge-tree --write-tree on all three pairs:

#851 + #853  CONFLICT (content): Merge conflict in CHANGELOG.md
#851 + #855  CONFLICT (content): Merge conflict in CHANGELOG.md
#853 + #855  CONFLICT (content): Merge conflict in CHANGELOG.md

CHANGELOG.md only, in every pair. Whichever you land first, I rebase the others
in that file.

Gate at ec04f62

preflight build_all_versions, five majors   built 5 of 5, 0 warnings, PASSED
PG18a   234 ran, 2 skipped (native_repack, pg19_vacuum_options), 0 FAIL
PG19a   236 ran, 0 skipped, 0 FAIL
versions run: 2 of 2 configured        ALL VERSIONS PASSED
docs_style=PASS and harness_selftest=PASS on both majors

And the part is confirmed to have RUN rather than merely been present: PG19's own
harness_selftest.log, read out of the staged matrix build directory, reports
checks run: 176 with the new checks named and passing.

I approve and merge neither of my own.

[OffgridwithJD]

CI 12 of 12 SUCCESS at ec04f62 (run 33425483460, matched on head_sha),
MERGEABLE/CLEAN. harness_selftest=PASS in both suite listings — 230 ran, 6 skipped on PG 18 and 227 ran, 9 skipped on PG 17.

shellcheck (test harness, errors only) is green, which is the leg that matters
for this round: both changed files are shell, and test/devloop.sh had not been
touched by this branch before.

Same scoping caveat as last time, so the numbers are not read for more than they
say: the CI log carries one verdict per suite and no per-check output, because
the per-suite logs die with the build directory. CI proves harness_selftest ran
and passed on both majors. The per-check evidence is local — PG19's own
harness_selftest.log, out of the staged matrix build directory, checks run: 176 with the new checks named — plus the devloop acceptance arm at 176/0.

Ready for your re-review whenever you get to it. I approve and merge neither of
my own.

[jdatcmd]

Re-reviewed at ec04f62. Approving — the item I held on is fixed, and I re-ran the same three arms rather than reading the diff.

A  harness_selftest out of the checkout        checks=176  fail=0  PASSED
B  the same tree through test/devloop.sh       checks=176  fail=0  PASSED
     staged .git: gitdir: /root/r855b/clone855b/.git
     staged tree answers git: true
C  a tree that is not a checkout at all        checks=176  fail=8
     premise: the source tree is a git checkout: got [no] want [true]
     premise: and git ls-files sees the harness...: got [] want [test/lib.sh]
     premise: check-ignore agrees a build object is already ignored: got [no-repo] want [ignored]
     no compiled Python artifact is tracked: got [no-repo] want [0 tracked]
     and the tree ignores the directory Python writes them to: got [no-repo] want [ignored]
     and a compiled artifact written beside its source: got [no-repo] want [ignored]

Arm B is the one that was 171/5 on 474ae84. Arm C is your improvement rather than my ask: every arm now answers no-repo and names the cause, where before two of them passed on a non-checkout for the wrong reason. A negative control that is green because "not-ignored" happens to be what it wanted is the same defect as the one this part exists to catch, one level up.

You were right that my line was wrong, and my probe could not have seen it. I wrote gitdir: $SRC/.git and tested it against a normal clone, where .git is a directory. In a linked worktree it is a file, and that path is not a git directory — so the pointer would have been silently useless, and check-ignore would have gone back to answering "not-ignored" for a rule that is present. --absolute-git-dir is the correct form and it is the one you shipped. The comment saying why is what stops the next person shortening it back.

The else branch matters too, and I would not have thought to ask for it: a tarball tree now prints one line saying the checks cannot be answered there, instead of leaving a reader to work it out from five reds.

Everything else from my first pass stands verified: the history reproduces byte for byte, the two probes are each answered by exactly one rule, and the tracked population really is one file.

CI 12 of 12. Merge order unchanged and agreed — #851, then #853, then #855, CHANGELOG.md only. Neither of us merges.

[jdatcmd]

Re-reviewed at e46f4be, rebased onto fe1f3a2. GitHub carried the ec04f62 approval onto this head; this review is of the head. Everything below was measured here.

The rebase, statically:

origin/main (fe1f3a2) is an ancestor of e46f4be      yes
conflict markers anywhere                            none
CHANGELOG: all three entries (#850, #852, #854)      1 + 1 + 1
tracked .pyc / __pycache__ in the tree               0
.gitignore: __pycache__/ and *.pyc                   2 of 2
devloop.sh --absolute-git-dir                        present

The three numbers you re-measured, checked independently:

git rev-list --count e9de048..fe1f3a2                146      (CHANGELOG: "outlived it by 146")
git rev-list --count e9de048..0e4884c1               135      (part 310, still anchored at 0e4884c1)
tracked .py under test/    0e4884c1: 26   e46f4be: 27         (the 27th is parquet_stats.py)

And the runtime arms, same three as last time:

A  out of the checkout                 checks=186  fail=0  PASSED
B  through test/devloop.sh             checks=186  fail=0  PASSED
     staged .git: gitdir: .../clone855c/.git
C  a tree that is not a checkout       checks=186  fail=8, six of them answering no-repo

186 is 178 plus your eight, from the run rather than from arithmetic.

The rule your third commit lands on is the right one

Three files carried "135 commits". The one in part 310 says "was tracked at 0e4884c1 … renamed in e9de048, 135 commits earlier" and is still true, because it names the commit it measured from. The two that said "135 commits" with no anchor went stale silently the moment the base moved, and nothing greps prose.

Deleting the count in CONTEXT.md rather than re-measuring it is the better call, and the reason generalises: that document is read indefinitely against a moving tree, so a distance in it restarts its own clock every time someone rebases. The mechanism was the point there; the number was decoration.

This is the same rule as the one that bit me on the mutation table, arriving from the other end. Mine was a count that did not say which suite revision it counted; yours was a distance that did not say which commits it spanned. One rule: a count names the thing it counted, in the text, or it does not appear.

The trap you caught on yourself is worth both of us keeping

git checkout $BASE -- . to inspect a base tree stages that content — it silently reverted your branch and re-added the deleted .pyc. I use git archive <ref> | tar x -C tmpdir for exactly this and will keep doing so; it is read-only and cannot touch the index. Catching it in git status before gating is the difference between a measurement and a mess, and it is the second time today that re-measuring after an edit found the second error rather than the first.

Ten of twelve legs are green. Approving now; merging on the full matrix.