Skip to content

lessons.md

Latest commit

 

History

History
604 lines (572 loc) · 42.8 KB

lessons.md

File metadata and controls

604 lines (572 loc) · 42.8 KB
Raw

Lessons

Rules for future sessions in this repo, captured after corrections and mistakes.

Process / workflow (user direction, 2026-10-15)

  • Always migrate completed items from todo.md to completed.md when a round closes. todo.md holds only the active plan; finished rounds live in completed.md (append a # Round N: … record with the items, outcomes, verification, and a Review section). Do it as part of closing the round, not "later".
  • Every round's plan ends with an item to re-evaluate improvements.md. A round is not complete until improvements.md carries the round's measured gains (new section + date) and that checklist item is ticked.
  • Push policy (revised 2026-10-15): commits/pushes to GitHub are allowed when the user asks, but never trigger GitHub Actions/CI (they cost money). Put [skip ci] in every pushed commit message. Only .github/workflows/test.yml runs on push (docker-release.yml runs on releases only); [skip ci] suppresses it.

Working files

  • All temp/working files go inside the morpheus folder (tmp/, git-ignored) — never in ~/.lmstudio/scratchpads. (User correction, 2026-10-07.)

Shell / tooling

  • The stemlib noun sources are named nom01…nom36 (no dot): the glob stemsrc/*.nom* silently matches only lsj.nom, nom.biblical, … Use stemsrc/nom* when searching all of them.
  • zsh chokes on unquoted parentheses in command lists; quote patterns or use a script file for anything with ( ).
  • zsh equals-expansion breaks unquoted =… tokens: echo ===MARKER=== dies with "not found" (zsh tries to expand =word as a command path). Quote every literal marker in shell one-liners.
  • When scrubbing a name from the repo, grep case-insensitively with word boundaries over ALL file types — not just .py/.md/.sh. Build artifacts (e.g. stemlib/Greek/conjfile, generated by concatenating stemsrc), generated reports, and saved snapshots all retain old strings long after the sources are clean. Fix at the root: edit the source, regenerate the artifact, delete stale snapshots.
  • Isolate a code change from snapshot drift with a stash A/B run (Round 8): when a saved evaluation snapshot no longer matches after your change, re-run it with the change git stashed — if the stashed run drifts identically, the difference is environmental/index noise (see endtable lesson), not yours; diff the two runs against each other to prove neutrality.

Beta code / Morpheus internals

  • cruncher only matches lowercase beta code; uppercase input (reference- corpus style) silently returns nothing. The toolkit now normalizes this (runner._normalize_beta); keep it that way and test both cases.

  • to_beta() drops a breathing when the Unicode letter has none: plain Ι gives *israhl, but Ἰ (U+1F08) gives *)israh/l. When verifying name entries, test with properly breathed/accented forms AND check -n (accent-insensitive) mode.

  • Explicit :wd: noun/name forms match exactly (diacritics included); the accent-insensitive pass (cruncher -S -n) is what makes unaccented variants hit.

  • Generative :no:<stem> os_ou … patterns do not work for every stem shape (worked for σάββατον, produced nothing for τρυβλίον/ἀδελφιδός). Verify each new entry against the built index (steminds/nomind) and fall back to explicit :wd: lines.

  • The incremental stemlib makefile does not always pick up a brand-new stemsrc file; if entries are missing from nomind/vbind, check timestamps and force the target (e.g. touch stemsrc/lsj.nom && make).

  • Paradigm declarations (;xx, @ xx) in vbs sources never reach the built index. The build is cat … > conjfile; do_conj; mv conjfile.short /tmp/vbmorph; indexvbs — indexvbs indexes the do_conj output, and only processes :vs/:aj/:no/:vb/:wd/:de: lines (all ;, @, --prefixed lines are ignored). So ;pp/;pf declarations change nothing in vbind; use explicit :vb: lines.

  • The generative deriv path rejects non-reduplicated perfect stems (derivio.c checkcomderiv2: if (!had_redupl && Is_perfect(stemtype)) continue;), so -έω pp participles (εὐλογημένος) never generate. Explicit :vb: lines have no such gate.

  • GenIrregForm requires a real stemtype in the keys: o_stem alone is derivtypes-only and fails with "no stemtype seen"; add e.g. omi_pr before it (precedent: existing :vb:di/dw omi_pr …). FixRecAcc/FixPersAcc preserve any accent already present, so a :vb:/:wd: line storing the corpus's exact surface spelling round-trips exactly through CheckGenWords.

  • standword() rewrites GRAVE→ACUTE (every position) and strips accents after the first on every input before lookup; CheckGenWords compares accents exactly in normal mode (morphstrcmp, only |≡i equivalence) but stripaccs both sides under IGNORE_ACCENTS. Consequence: a stored word-final grave can never match an accented input — store the acute spelling instead (or both spellings, as with αλληλουία). The reference corpus is internally inconsistent (e.g. νοσσιὰν and νοσσιάν for one form; λημφθήτω unaccented in beta but accented in its Greek column) — index every attested standardized spelling.

  • Noun lookup (chckindecl) always strips all accents from the key; nomind keys are accent-stripped and merge all full-form variants sharing a stripped key onto one line (morphstrcmp: |≡i, plus "s1 ends ⇒ equal" quirk). The lookup is therefore accent-blind; only CheckGenWords' final compare is accent-sensitive. Multi-entry lines are fine — the parser iterates all entries.

  • C# port has placeholder stubs (chckindecl.cs returns 0; CheckGenWordsFunc uses plain string.Compare, no comptab/accent logic) — don't assume C/C# parity when reasoning about behavior from the .cs files.

  • nomind key/entry mechanics (Round 7, cost hours): index lines are <key> <form>:<lemma>:<class>[:features]…; lookup keys carry a class prefix (2=indecl/nom, 3=deriv); the pre-index is <file>.lindex (fixed 12-byte records: 8-byte key + 4-byte LE offset). A :wd:<form> line whose form has NO accent marker gets its form field DROPPED by indexnoms (entry becomes <key> :<lemma>:<class>); forms with / etc. are preserved.

  • The real gate is CheckGenWords: generated forms must match the input word; for non-enclitics the compare is accent-exact (accword), and FixPersAcc ADDS an accent to one-syllable stems — so unaccented inputs (elided forms) only match under IGNORE_ACCENTS (-n). That's why nom.elided entries resolve only with -n, leaving default-mode behavior untouched.

  • C build on this Mac: link against src/{gener,gkends,gkdict,morphlib, greeklib,anal}/*.a; needs the installed Xcode app's SDK exported as SDKROOT

    • -isysroot $SDKROOT (the CommandLineTools SDK is broken — same workaround as scripts/build.sh). C binaries resolve data via env MORPHLIB=/stemlib and append /Greek/… themselves; run indexnoms with that env set.

  • :vb: stemtype tokens must come from rule_files/stemtypes.table, not derivtypes.table (Round 8, cost an hour): the analyzer's key parser (ScanAsciiKeys in morphkeys.c) loads STEMTYPES = stemtypes.table; verb types that exist only in derivtypes.table (anw, airw, …) are silently indexed but fail at analysis time with "GenIrregForm Error: no stemtype seen". The stored full form doubles as the generation stem, so ANY accepted token works — the generator's identity slot reproduces it and CheckGenWords matches (Round-8 vbs.papyri uses emi_aor / ow_pr+o_stem / ew_pr for this reason). Reproduce with: bin/cruncher -S -n <file> on a one-word file — the error prints to stderr.

  • Endtable/deriv regeneration is not byte-stable across builds (Round 8): after any stemlib index rebuild, saved AF/LXX snapshots can drift ~±0.5pp POS while top-1 stays identical — even when rebuilding from pristine sources. Before chasing a snapshot "regression", re-run the evaluator on a clean-source rebuild to separate build noise from your own changes; then refresh the snapshot and note it in the round record. (Round 7's byte-identical result held because no index rebuild happened that round.)

  • cruncher <name> reads <name>.words, not <name> — name probe files accordingly or it reports "cannot find".

  • Latin lemmas carry #N homonym markers that survived normalization (Round 8): the Latin stemlib disambiguates irregulars as sum#1, edo#1, quis#2; the parser's trailing-digit strip left the bare # in Analysis.lemma, so frequency lookups for the most common words in Latin (est→sum) silently scored zero. _TRAILING_HOMONYM is now [\d#]*$ (Greek lemmas contain neither — verified zero occurrences, A/B AF re-run byte-identical). Related: Morpheus's Latin analyzer cannot separate adjectives from nouns (both surface as N unless a special stem type fires), so never build joint lemma×POS tables for Latin from external gold that tags them separately — the Ranker's known-not-seen floor would demote correct readings.

  • Word-final U+0313 is the classical elision mark, not a letter part (Round 9): AGLDT Greek marks elision with a combining comma above on the final character (δ̓ = δέ); to_beta() used to reject such forms outright — 34K tokens, most of them in Homer, silently unanalyzable. Map it to the ASCII apostrophe Morpheus already recognizes, but ONLY at end of word: mid-word combining marks are letter parts (ϐ, digraphs) and must pass through. Before touching shared conversion code, scan every committed gold corpus for the mark — SBLGNT/AF/ PapyGreek had zero word-final occurrences, which made the fix provably neutral.

  • runner.analyze_beta() takes BETA CODE, not Unicode (Round 9): the API layer converts via to_beta first; _normalize_beta only lower-cases. An evaluator that feeds raw Unicode gets a silent 0% coverage (cruncher matches nothing, returncode still 0). In -S mode matched forms print to stdout and unmatched ones to stderr — probe with -q, or check both streams.

  • Beta-code accent markers follow their vowel, so hand-rolled betas are wrong (Round 9): ὡς is w(s, not ws/; the marker attaches to the accented letter, and a mistyped beta looks exactly like "form not found". Always generate probe strings via to_beta() (or copy an existing table key) instead of typing them.

Corpus tooling (annotation pipelines)

  • Short beta forms collide with real words under accent-insensitive matching (Round 9): letter numerals like κε/κα/ιε match actual stems in -n mode (κε→ἄν, ιε→εἶμι, ιη→κερατέα — 315 of PapyGreek's 348 "analyzed" letter-numeral hits carry the wrong top-1), and conversely real one/two-letter particles exist in other corpora. So synthetic-analysis features must fire only on UNANALYZED tokens, never override analyzed ones; and "covered" ≠ correct — always check top-1 on the covered set before celebrating a coverage gain.
  • PapyGreek ALDT specifics (Round 8): postag is 9 chars — pos person number tense mood voice GENDER CASE degree; gender precedes case (opposite of MorphGNT's order) and letters are lowercase. The editorial-marker zoo beyond LappEpiDoc brackets: ∼∽∤⋰⋱⸢⸣❨❩<>[]{}() + U+0324, |num:N|/|g:…| pipe spans, plus 〚〛 ⊢⊣ ⸌⸍ ⎴⎵ ⦑⦒ ⸜⸝ ⧼ ? … -. Missing any one silently corrupts forms (fake misses); keep ʼ ’ (real elision) and combining diacritics.
  • Morpheus.analyze_beta() returns TokenResult wrappers, not raw analysis lists (result.analyses). The runner-level method returns bare lists; the API level does not.
  • Reference-corpus parse codes are position-dependent by lexical class: verb codes are 5 chars (tense voice mood person number), nominal codes only 3 (case number gender). Any field-by-field comparison must branch on class AND use per-class minimum lengths, or most nominals silently drop out of the comparison.
  • Morpheus's 8-char parsing code contains dashes (3AAI-S--), so shape-based column detection ("parse = the dash-free [A-Z0-9]+ field") misparses generated files — it reads the parse code as the lemma. Parse generated output positionally.
  • When training disambiguation models on a gold corpus and then evaluating agreement with that same corpus, say so in the report: the numbers measure domain consistency, not absolute accuracy.
  • Never hardcode the reference-corpus location (path or name) in any file. Scripts take --corpus-dir / env (LXX_CORPUS_DIR, AF_CORPUS_DIR); generated reports and bundled tables use a neutral label ("Apostolic Fathers reference corpus", LXX precedent). The user greps the working tree — committed files, tmp/ scratch, and build artifacts must all stay clean.
  • A lemma-keyed convention table needs an M==0 guard, not just single-target + frequency (Round 7): form-dependent splits — gender pairs (κύριος/κυρία), enclitic compounds (ἐγώ/κἀγώ) — pass "all mismatching tokens map to one target" but rewriting every occurrence of the source lemma breaks tokens where Morpheus and the gold already agree. Require that no trustworthy token with that choice agrees with the gold, and print rejected pairs (with reasons) in the summary so the table stays auditable.
  • Viterbi vs top-1 is a real trade on AF (Round 7): the tagger costs ~1.4pp of lemma (MorphGNT subset) for +1.0pp POS — unlike LXX, where it was net positive. For corpora whose primary column is lemma, default to top-1 and expose --tagger (matching agreement_report.py's flag semantics).
  • After toolkit changes, re-run saved evaluators and diff against snapshots (Round 7): the matches_lemma hardening normalized gold that callers already pre-normalized — the ev2 snapshots came back byte-identical, proving no behavioral drift. Keep snapshot files in tmp/ for exactly this purpose.
  • When porting a pipeline across repos, diff outputs as the test (Round 7): the AF repo's analyze.py mirrors morpheus's build_morph_af.py; all 15 generated files verified byte-identical before shipping. In the consuming repo, derive tables at run time from its own gold (no stale bundled copies) and pin the minimum upstream commit in the README.
  • The reference corpus is UPPERCASE beta code with its own dialect quirks (Round 6): breathings follow the letter (A)PE..., not APE...), / is a plain syllable separator inside words (KURI/OU = κυρίου — substring greps like RIOU miss it), and \, =, | are accent markers used inconsistently for the same word (αὐτοῦ appears as AU)TOU=, AU)TOU\S, AU)TOU/S). Never assume a spelling: key everything through canon() (to_beta∘from_beta + lowercase), and remember one word can occupy several canon keys that must all be covered by any table.
  • The eval's "top lemma disagreements" list is misleading (Round 6): it sorts by total form frequency and prints only the first disagreeing line, so a rare quirk on a frequent form looks like a mass error (ὅτι read as ×4,044; actually 7/4044). Always verify with true per-form mismatch counts before sizing a "prize pool".
  • Joint-frequency fallback: score unattested (lemma, POS) pairs at a floor of one occurrence — never zero, never full plain frequency (Round 6): the reference's type scheme and Morpheus's POS disagree for the same lemma (αὐτός A vs RD), so zero kills correct candidates; full plain frequency over-credits them (σός(N) 0.432 vs σύ(RP) 0.838 left a gap bigram context flips). An unattested pair can never be the gold reading of any token, so the floor only removes wrong answers.
  • The Viterbi tagger keeps one candidate per coarse POS (first on score ties) (Round 6): feature-level ambiguity within a POS — masc/neut homographs, duals — is resolved arbitrarily, and candidates with empty parse codes pass field comparison vacuously (no comparable pairs ⇒ True). Break same-POS ties explicitly (per-spelling reference aggregates; the corpus's accent notation encodes readings) instead of trusting Morpheus's internal candidate order.
  • Analysis.lemma/.variants() are Unicode, frequency-table keys normalized Unicode (Round 6): compare via normalize_lemma(), never raw literal equality — from_beta can return NFD and precomposed Greek literals won't match it.
  • PROIEL CoNLL-X quirks (Round 8): Greek gold lemmas carry #1 homonym suffixes and optional-letter parens (οὕτω(ς)) — normalize both before comparison or you overstate mismatches (~2.5pp artifact). Latin Morpheus lemmas carry bare # markers — strip #\d*$. CoNLL-X exports also omit unannotated sections (greek-nt: HEB 13 + Catholic epistles; 132,366 of 140,773 XML tokens).
  • Decompose residual mismatches into rankable vs convention before sizing a learned model's payoff (Round 9): of the 644 homograph-table forms, only 42 had non-degenerate gold marginals and just 13 any context split — Koine gold is ~95–100% one-way for most ambiguous forms. The famous "residuals" (ὡς/ὅς, εἰ/εἰμί) are unanimous in the training corpora; their disagreement with PROIEL is inter-edition annotation convention a ranker correctly doesn't chase. Measured gain was +0.6–1.6pp, not the +2–4pp target — that is the ceiling of what context can buy here, not an implementation bug.
  • PROIEL lemma conventions differ from SBLGNT/Morpheus (Round 8): pronoun families headed by the plural (ὑμεῖς/ἡμεῖς vs σύ/ἐγώ), εἶπον kept under λέγω, lowercase σατανᾶς, iota Ἡλίας where SBLGNT has eta Ἠλίας. Cross-corpus "errors" must be split convention-vs-true before sizing fixes (~3,100 tokens ≈ 2.4pp of the PROIEL gap were pure convention).
  • License footers lie (Round 8): the Digital Classicist wiki footer said CC BY for the Index Thomisticus Treebank; the project site states CC BY-NC-SA 3.0. Always check the authoritative project page, not aggregator wikis — it decided between two candidate Latin table sources.
  • AGLDT treebank format (Round 8): words are <word form=… lemma=… postag=9chars/> elements; lemmas carry Perseus-style trailing homonym digits (for1) which normalize_lemma() already strips. Latin tagset position 1: r=adposition, p=pronoun (opposite of PROIEL UPOS R/P); Greek v2.1 uses Unicode lemmas with initial capitals (normalized lower-case). One reader (--format agldt) serves both languages; filter non-letter keys (-que, heu!, bracketed restorations) and skip postag u.
  • Private corpora (TLG / PHI) are measurement-only — commit nothing derived. TLG and the PHI CD-ROMs (Latin, and the Greek inscriptions + documentary papyri) are all-rights-reserved / scholarly-use-only — stricter than PROIEL (CC BY-NC-SA). They are also untagged, so they can serve neither as gold nor as a shippable frequency table; their only use is coverage probing from tmp/ (reuse the PapyGreek marker-stripping). Never commit their text, miss lists, or any normalization/frequency table; add only generic entries justified by public lexica (LSJ/Bechtel). Open equivalents (First1KGreek / Open Greek and Latin, CC BY-SA) are commit-safe for the same job. Recorded in todo.md "Round X".

Python

  • Augmented assignment on a function call is a SyntaxError: d.setdefault(k, set()) |= x — use d[k] |= x with a defaultdict instead.
  • When writing generators that emit build inputs (stemlib files), make re-runs idempotent by merging with the existing output; otherwise entries generated in run N are dropped in run N+1 once the tool's own output becomes "recognized" input. Use an explicit managed-section marker so curated content is never rewritten.
  • Managed-section trap (hit for real, Round 2): a generator that rewrites its whole output file from its own lists will silently wipe anything hand-added above the marker. Keep all curated entries in the script's data structures — never hand-edit the managed region of the generated file. When normalizing emitted values (e.g. grave→acute), apply the normalization at both entry points (new data and load_existing() merge) or re-runs won't converge.

Round 10 (2026-10-15)

  • The stemlib build IS byte-deterministic within one environment — two make clean rebuilds produced identical steminds/, endtables/, derivs/ (equal hash, equal to HEAD). The ±0.5pp snapshot drift the earlier rounds saw is cross-environment, not run-to-run: the only make clean deltas are the compiled STEMPROG helper binaries (stemsrc/{stripref,zapfirstf,headnolen,goodstem,flatlems}), whose bytes embed build metadata and do not affect analysis. Before blaming "build noise" for a snapshot move, verify determinism locally (rebuild twice, hash) rather than assuming it.
  • $(wildcard …) order is filesystem-dependent — the Greek makefile fed stemsrc/nom.* to indexnoms in readdir order, a cross-environment hazard. Wrap in $(sort …) for a stable build input; verify neutrality with a forced rebuild (touch stemsrc/lsj.nom && scripts/build-stemlib.sh) and git status.
  • The aorist passive of -έλλω verbs was never generated, so simplex ἠγγέλη and compound ἀπηγγέλη silently fell back to the unrelated real lemma ἐγγελάω (ἀπό+ἐγγελάω). Explicit :vb: lines with stemtype aor_pass (stemlib/Greek/stemsrc/vbs.r10) fix the simplex and the compounds — the preverb split (unasp_preverb) then yields the correct lemma ἀπαγγέλλω. A "compound-decomp bug" was really a missing-paradigm gap; check simplex forms first before blaming the compound scanner.
  • Interjections analyze with stemtype exclam, not interj — fine_pos() must map exclam→I (οἴμοι/φεῦ/ἰού), or they surface as N.
  • Learned re-ranker: morphological-feature tokens are what generalize; neighbour lemma context overfits (predicted-context noise): the POS+stem+morphology model (488 features) beat the version that added closed-class neighbour lemmas on every independent corpus despite lower train error.

Round 10, Items 10–13 (2026-10-15)

  • Reported confidence was miscalibrated, not just "a softmax". Ranker scores are ordering scores; the raw softmax was under-confident (ECE 0.21). Fit a single temperature by NLL on covered gold (scripts/calibrate.py) — the ratio matters, and the fit transfers (AF ECE 0.20→0.14). Compute ECE / top-1 accuracy conditional on coverage when calibrating, or the accuracy looks impossibly high (raw top-1 sits below coverage).
  • "Dictionary forms" are mostly already produced by Unicode conversion. Morpheus's beta article lemma o( becomes ὁ via from_beta, and pronouns/demonstratives already come back as ἐγώ/σύ/οὗτος/ἐκεῖνος. The LXX driver's --dictionary-forms gain is a reference-format/POS-column matter, not a lemma rewrite — a static "profile" transform for the toolkit is therefore (correctly) idempotent.
  • Parallelism is trivial and effective here because each job is a cruncher subprocess (I/O-bound): ThreadPoolExecutor over contiguous chunks gives 3.06× at jobs=4 with identical output — no pickling/multiprocessing needed.
  • Local wheel build needs setuptools in the venv; pip wheel --no-build-isolation then works offline (the cached wheel suffices). Reference data travels in the wheel (include-package-data).

Round 10, Item 5 (-πλ artifact) (2026-10-15)

  • from_beta decodes -pl as -πλ (p=π, l=λ), so the artifact shows up as a Greek suffix (Σατάν-πλ), not ASCII -pl. Grepping the Unicode output for -pl finds nothing; search for πλ.
  • addconstraints.pl's :le:<lemma>-pl group entries were pure aliases: the engine generates every number from one stem+ending class, so a pl-tagged group entry still produced the singular surface forms and made them match the bogus lemma. Emitting the group entries under the base lemma (:le:$curlemma) removes the alias, keeps coverage (the base entry already covers the plural), and lets indexnoms merge the duplicates.
  • Effect size: a correctness fix, not an accuracy win at corpus scale — 4/40K SBLGNT and 26/120K AGLDT top-1 lemmas carried -πλ; both drop to 0, and the affected names then match gold when the base lemma is gold-correct. Corpus aggregates stay within the ±0.5pp build-noise band, so it will not show in the benchmark — measure the artifact directly (scan for the suffix), not via aggregate accuracy.

Round 11 (2026-10-15)

  • Pooling a large training corpus without balancing overfits the larger domain. Adding all of AGLDT (468K tokens) to SBLGNT+AF (200K) made the learned re-ranker much better on classical (+3.2pp over the Koine-only model) but worse on Koine (SBLGNT −0.7, AF −0.6, PROIEL −0.4). --agldt-limit 200000 (uniform subsample, ~50/50) kept almost all the classical gain (+3.0) with Koine within the ±0.5pp build-noise band (−0.3). Always measure a pooled model on every domain, not just the target — the benchmark gate only guards coverage/ranked/POS, so a learned regression slips through unless checked explicitly.
  • AGLDT *.tb.xml files are huge single context chunks. Analyze in batches (4000); feeding a whole Homer file (~200K tokens) to one cruncher call times out. This bit the trainer and the benchmark harness both.
  • The POS-bigram Viterbi tagger still loses to frequency-ranked POS, even with an in-domain classical bigram model (AGLDT 85.2% → 83.6% tagger). The classical bigrams do beat the Koine ones for the tagger on classical text (Homer 87.3 vs 84.8), but the conclusion stands: a learned-emission model is the real requirement, not more bigram data.
  • treebank_data beyond v2.1 is redundant or negligible: AGDT2 = Aesop + P.-Apollodorus (already in v2.1), v2.0/v1.6/v1 = older copies, IGDS = ~73 words. Harvest v2.1 only.

Round 13 (2026-10-15)

  • --ignore-accents ignores accents, not breathings — and confidence does not save the miss. stripacc removes acute/grave/circumflex only. The breathing retry (smooth, then rough, stdiomorph.c) runs only when checkstring misses entirely. Bare en is a miss without -n. With -n it hits epic ἔν (e/n,ei)mi/, imperfect 3pl), the only candidate, so the API reports confidence 1.0 — not P ἐν. e)n is ἐν at 0.99. Without the flag, recovery proposes εἰ (0.49), not εἰμί; the published example is true only of the flag. Do not recommend --ignore-accents plus confidence as the uncial fix. "100% analyzed" on John 1:1–2 is coverage of 24 tokens, not lemma accuracy. Corrected 2026-10-16; the Round 13 demo/email had the mechanism wrong.

Round 15 (2026-10-15)

  • A measurement harness must use the shipped rankers for its baselines. The B8 proto's ad-hoc frequency order (hand-rolled freqs.get(normalize_lemma(...)) sort) gave PROIEL freq 92.7 where the shipped Ranker gives 95.4 on the same candidate sets; the head-to-head's corpus-level projections reproduce benchmark.py exactly (AF 90.1/90.3, PROIEL 91.7/92.2), which is the check that makes the comparison trustworthy. Reuse the toolkit's own code paths in evals.
  • Measure before fixing a suspected miscalibration. The learned flip made the bundled T=0.08 (fitted on frequency scores) look wrong for learned scores; the probe showed the learned softmax is natively calibrated and insensitive to T in [0.08, 1.0] (ECE ~0.005 throughout). The per-path fit (T=1.0) is still the correct engineering, but the expected numerical effect was near zero — the real win is the documented 0.142 → 0.005 default-path ECE.
  • Default flips need a "never worse" measurement on every corpus — including the candidate configuration, not just the flag. Homograph re-ranking measured +0.2 to +0.9 on SBLGNT/AF/PROIEL but −0.4 on AGLDT classical, so it stays opt-in. The learned path also gives back ~0.2pp of the AF table-selection gain because its features are trained against the default table — a table-selected ranker is not automatically better for a learned re-ranker on top of it.
  • Corpus-level adaptive selection must batch. select_adaptive_table analyzed group-by-group (SBLGNT verses, PROIEL sentences) — one cruncher spawn per group, turning a 20K-token sample into thousands of process spawns. Flatten the sample and analyze in 4000-token batches.
  • The LM disambiguation layer is distribution-sensitive. Ancient-Greek-BERT + MLP head wins on accented text (SBLGNT test 99.4 vs 97.9 learned, ambiguous-only) but collapses on the shipped accent-insensitive pipeline (PROIEL 92.3 vs 96.4): ignore_accents inflates multi-candidate tokens 37.8% → 59.4%, and the head was trained on accented gold only. Candidate generation defines the task; train the scorer on the candidates the toolkit actually produces.

Round 16 (2026-10-15)

  • The cruncher's accent-insensitive mode does not ignore the iota subscript. normalize() strips the combining ypogegrammeni (U+0345 is a combining mark), so every generated variant of a ῃ form loses the subscript and can never match the paradigm entry the stemlib keeps as h| — verified directly: u(giai/nhs → 0 analyses, u(giai/nh|s → ὑγιαίνω. Documentary texts drop the subscript constantly, so recovery restores it on single η positions (ὑγειαίνῃς → υγιαινῃς → ὑγιαίνω); without that, tier-1 variants of such forms are all dead and the tier-2 combination that would revive them is beyond the cap.
  • Reproduce a suspected pipeline bug through the exact code path before "fixing" it. A miss-list reading suggested ὑγιαίνιν was skipped by the recovery pass; calling the shared recover() helper directly (same runner, same table) recovered it — the list had been misread. One direct call is the cheap falsification step.
  • When the measurement harness changes, re-measure the baseline under the same change. Stripping |num:N|/|gap:N| spans from gold lemmas moved the PapyGreek baseline learned top-1 from 87.8/89.3 to 87.9/89.4. Comparing the new run against the old baseline would have misattributed +0.1pp to the feature under test.
  • Additive coverage gains move coarse POS the "wrong" way, by construction. Recovery fires only on unanalyzed tokens, so coverage and top-1 can only rise — but POS is measured over analyzed tokens, so its denominator grows with tokens that agree less often: PapyGreek POS 80.1% → 79.7% (the ~1,485 newly scored tokens agree at ~70%) and, on the gate, AF 75.94% → 75.31% (analysis rate 98.37% → 99.4%; the ~650 new tokens agree at ~15%). No analyzed token's analysis or order changes, and the arithmetic reproduces both drops exactly. Document this as a completeness artifact; do not "fix" the metric by excluding recovered tokens.

Round 17 (2026-10-15)

  • Generate variants from primitive operations on the normalized base, not from position slots. Round 16 paired slot objects for tier 2, which could not combine a substitution with a geminate or a global rule, and made stale positions easy. Round 17's _edits emits each operation as an edit list on the base, so merging two operations is concatenation, positions never go stale, and pair eligibility is a per-operation flag — the same shape also supports the pair-tier ordering (vowel substitutions first) that keeps the dominant documentary pattern closest to the front. θρυδάκεια → θριδάκια (υ→ι + ει→ι) and ὑγειαίνησς → ὑγιαίνῃς (ει→ι + σσ→σ) became reachable.
  • Measure a symmetric rule's cost before shipping it. The ε/η and α/ο interchanges are textbook Koine confusions, but the experiment over the actual miss list fixed 3 and 6 tokens; both add tier-1 operations that shift every existing target deeper into the capped list. At 9 tokens total they are deferred with the evidence recorded — the interchange classes stay a candidate for a round that also measures cap pressure.
  • Read the corpus builder's filters before suspecting the data. χμγ (×3) was absent from the numeral table despite meeting the min-count: the builder keeps only gold pos m tokens, and χμγ's gold pos is x. The compositional vowel-less rule covers it (and five more forms) without touching the table.
  • The gate's lemma comparison is not the harness's. benchmark.py scores a hit when ANY lemma variant matches (the standing decision), while the tmp PG harness compares the top analysis's canonical lemma — on PapyGreek the same run reads 87.7 ranked (gate) vs 85.9/87.3 exact/acc-insens (harness), and the learned gap is similar. Quote harness numbers in round records; use the gate as a threshold only. The POS classes must likewise invert the harness's TB2CLASS/MORPH2CLASS maps exactly — the DC bridge (C↔D) and the X/I equivalence included; the first implementation omitted both and read 79.8 where the harness reads 81.3.
  • A confirmation run is a fresh run. Round 16's gate confirmation was launched in the background and did not survive the session boundary; the record had been written as if it were pending confirmation and had to be softened afterwards. Re-launch a confirmation in the session that will record its result, or state explicitly that it was not run.
  • Proposal confidences are conditional on the generated candidate set. _annotate softmaxes over the token's current candidates, so a lone proposal reports 1.0 unconditionally and all-proposed tokens average ~0.84 on PapyGreek while the measured top-1 accuracy of recovered forms is ~0.59 — overconfident by construction, because the candidate set is generated rather than exhaustive. A fitted prior per proposal strategy belongs in calibrate.py; until then the actionable signal is the proposed/recovered_from flags (carried to B4, with the audit recorded in tmp/pg17/audit_17_5.md).

Round 18 (2026-10-15)

  • Score on the canonical lemma, never on the variant list. Morpheus's form,lemma candidate field carries the form element the stem matched, and an alternate stem can make one entry's form element coincide with a different word's lemma: πηρός's Doric stem lists παρά, so variant-list scoring credited πηρός with παρά's frequency (366) and put it above the actual preposition. Scoring normalize_lemma(analysis.lemma) (the last variant, which the parser already stores as lemma) fixed the class: PapyGreek plain path +1.3pp ranked (88.1 → 89.4 exact, 89.5 → 90.8 accent-insensitive), SBLGNT unchanged (94.6 ranked / 77.9 POS), +26 recovered-form hits. Variants stay for matching (gold corpora, the Strong's fallback) — matching wants recall, scoring wants identity.
  • A scoring primitive propagates into every consumer: fix it in one place, then re-verify every re-ranker. The same variant-list lookup lived in four places — Ranker.frequency_score, the learned model's _best_frequency/_best_joint, the adaptive table fit (tables.py) and the opt-in homograph re-ranker (homographs.py). The adaptive picks are load-bearing (they choose each corpus's frequency table), so the gate's per-corpus adaptive table: lines are the check that the fix did not move them.
  • Retrain on the shipped candidate distribution, or the model learns a task that does not exist at inference. The Round 15 trainer skipped tokens with no analysis; since Round 16 the shipped pipeline recovers them (spelling/name proposals), so the trainer now runs the same recovery pre-pass (411,615 analyzable tokens vs 385,758) and ranks proposals as part of the task. A document-level held-out split (79 PapyGreek documents, 6,936 analyzable tokens) keeps the PG number honest — same-document leakage would flatter it, and the trainer now reports the split in the model file itself (papygreek_holdout).
  • A confidence is conditional on its candidate set; generated candidates need a fitted prior. (Round 17's audit, closed in B4.) The class-mass POS estimate is peaked for the lemma, not for the class: SBLGNT mean mass ~1.0 against 79.4% coarse-POS accuracy (ECE 0.19). The reported POS confidence therefore multiplies the mass by a fitted per-class reliability, and all-proposed tokens are scaled by fitted per-strategy accuracies — a lone proposal reports its strategy's measured rate, not 1.0.
  • Verify a commit's diff, not its message. The B6 commit claimed to untrack the STEMPROG helper binaries, but git show --name-status showed M (a rebuild had been committed) and git ls-files still listed them — and .gitignore has no effect on tracked files, so make clean would still dirty git status. Caught on review and fixed by rebuilding the two commits (git reset --soft + re-stage): the helpers are now untracked (D) and ignored, and the makefile's STEMPROG target compiles them from stemsrc/*.c (the clean target removes them). Amending history also needs the right commit — --amend hits HEAD, so check git log -1 first.
  • Replicate the reference converter's algorithm, not your reading of the problem. (Round 19.4, the C Unicode path.) The beta-code library is exactly "NFC normalize, look every codepoint up in unicode_to_beta_code.json, pass unmapped characters through" — three lines of Python. The C port therefore generates every table from that same JSON (map, NFD expansions, composition pairs, combining classes, lowercase) and re-implements NFC, instead of hand-writing base-letter/mark rules. Parity is then by construction wherever the tables are the same data; the oracle only has to prove the NFC re-implementation.
  • NFC is full NFD then recomposition — singleton decompositions are not pairs. U+0340 → U+0300, U+0343 → U+0313, U+0344 → U+0308 U+0301 have canonical decompositions that never appear as two-codepoint composition pairs. A converter that only composes pairs passes them through unchanged and diverges (367 of 6,623 synthetic forms). The fix: expand every input codepoint to its full NFD first, then recompose.
  • Composition starters include decomposition pieces, including spacing characters. U+0385/U+1FC1/U+1FED-U+1FEF decompose to spacing diaeresis/backtick (+ mark), and NFC recomposes them from those pieces; a pair table built only over Greek-letter starters misses 16 forms. Derive the starter set from the NFD expansions themselves.
  • A generated C table with a fixed-size terminator needs the terminator's slot. int nfd[4] with initializers {c1, c2, c3, c4, 0} compiles with only a warning ("excess elements in array initializer") and drops the 0 — then the read loop walks into the next struct's memory and emits garbage (a 1,134-form parity explosion). Check the build log for warnings on generated tables; size arrays as max + 1.
  • "\x1fFAIL" is not \x1f + "FAIL" in C. Hex escapes are greedy: \x1fF is parsed as a hex value out of range. Split the literal ("\x1f" "FAIL") or use an octal escape (\037) where the next character is a hex digit.
  • An exhaustive oracle is cheap insurance for a "provably equivalent" port. The synthetic set (every mapped codepoint, every Greek codepoint, pair compositions in NFC and NFD spellings, mark orders, apostrophe variants, 3K random strings) plus every distinct corpus form (110,927 total) found the 367-form singleton gap, the 16-form spacing gap and the struct overflow — none of which spot checks on real words would have surfaced.
  • Audit under the shipped configuration, and validate the audit script against the recorded numbers first. (Round 19.1.) The PG error audit runs with the adaptive papyri table (the shipped pick); the default Koine table scores 85.6/88.2 vs the adaptive 88.1/89.6, a 2.5pp shift that would have changed the classification mix. Before trusting new classifications, the script reproduced the Round 18 record's 88.0/89.5 (numerals off) exactly.
  • Split "gold absent from candidates" from "gold present, not top-1" before proposing ranking work. Of 2,857 PG top-1 errors, 1,725 are gold-absent — and 1,533 of those are editorial/regularization conventions no ranking change can fix. Only the 1,132-token gold-present bucket is ranking-fixable, which bounds a context re-ranker's honest target to fractions of a point (+3.1pp ceiling acc-insensitive) before spending the training run.
  • A gated optional pass must not run before the heuristic that gates it. Adaptive numerals (numerals="auto") fire only when the adaptive selection picked the papyri table, so the numerals pass runs after _apply_adaptive_tables — otherwise numeral proposals would feed the table fit and could flip the selection that is supposed to gate them.
  • Prefer a distinct flag to an optional-value flag when a positional argument follows. --numerals-auto (separate store_true) instead of --numerals=auto (nargs="?"): the optional-value form is ambiguous to users — --numerals file.txt silently eats the path argument.
  • A smoke test that omits the held-out set never exercises the evaluation path. (Round 19.3.) The MLP trainer's first full run crashed in evaluate() (sum(len(...)) on an int) after ~40 minutes of preparation + perceptron training — the earlier 2K-token smoke test had no held-out examples, so evaluate was never called. Cover every code path the long run will hit with synthetic data first; the fix was verified in seconds with a fabricated held-out set.
  • Don't edit a script while a long background run is executing it. Python renders traceback source lines from the file on disk at crash time, so the interleaved edit made the traceback show wrong line numbers and text (frames pointing at with open(args.out) for a crash in evaluate). The misleading frames cost real debugging time; wait for the run or edit a copy.
  • A held-out gain below ~2 standard errors is not evidence. B8-lite's MLP beat the perceptron by +0.21pp on the 6,936-token PG held-out split — about 15 tokens, with SE ≈ 0.24pp — and the pre-set ≥0.3pp ship gate correctly declined it. Report the token count and noise scale next to the pp gain, and let the pre-set gate decide rather than post-hoc enthusiasm.
  • Reproducibility is a check, not a hope: retraining reproduced the shipped perceptron byte-identically. Diffing the retrained payload against the shipped model (weights, feature index, held-out numbers all equal) proved the training pipeline unchanged and the environment deterministic — and would have caught an accidental data or feature drift before it reached a model file.