You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Child of #2812 (Phase 4 · score contracts). First formula to get a score contract; the pattern here is meant to be reused for risk_api_exposure, risk_tech_debt and func_complexity_gini, the other score-layer rows still carrying open-defect cells.
Labels: epicmetricscore-engineroadmap
Why this epic exists (one paragraph)
_calc_documentation does not measure documentation. On the rosetta corpus every term but one is flat: no function has impact above 50 (the "opaque execution" term is dead), every file is under the 50-line evidence floor (the denominator is a constant 70), and the multipliers do not move the observed values. What remains is a sigmoid over 2 × api, so one adjusted-api hit per file is worth about 13 points on a 0–100 scale. The adjusted api count is assembled from two overlapping sources (the language's api rule plus the orphan-census credit, minus a tokenizer-dependent overlap), and every way that assembly fails lands in a different score cluster: 79 where a name is counted twice (m4, makefile, scheme, cobol), 55 where the rule sees one token and the census credits three (css, agc_assembly, yacc), 18 where the census is unanswerable or broken (dockerfile, html, yaml, haskell), 0 where the rule matches nothing (jcl, sqlite, objective-c main). On real code the same equation flips regime: a 500-line file with the same three undocumented public functions scores about 15, because the denominator is now the file length. That is the −0.83 length leak the bias report already flags, seen from the other side. Four of the eleven open-defect cells on the scoreboard and the corpus's single confirmed length leak are this one formula.
The goal restated:risk_documentation is the share of a file's callable surface a reader cannot recover from documentation. A ratio over units, not a density over lines. Identical intent scores identically by construction, not by tuning.
The contract
risk_documentation — of the units extracted from a file (functions, methods, paragraphs, steps), the weight-share that is undocumented. A unit is public when its name is in the file's public name set. A unit is documented when a doc-rule match is anchored to its header window. Public units weigh double; a unit's weight rises by one per reflection_metaprogramming hit inside it. A folder-level documentation umbrella shields the whole file multiplicatively. A file with no extracted units has no value (n/a), not zero.
for each extracted unit u:
weight(u) = (2 if public(u) else 1) + reflection_hits_inside(u)
exposed(u) = weight(u) if not documented(u) else 0
risk_documentation = 100 × Σ exposed(u) / Σ weight(u) × (1 − umbrella_shield × doc_umbrella)
= n/a when Σ weight(u) = 0
documented = a doc-rule match whose span ends inside the unit's header window: the k lines immediately before start_line, or the leading lines of the body (docstring position). Not the slicer's docstring field, which is a per-language "preceding comment" heuristic: on the corpus it captures python's planted # HACK: comment as a docstring, reads 13 of 13 for jcl, and 0 for fortran, haskell and scheme.
six knobs become two: public_weight (2.0), umbrella_shield (0.5)
Weights are counts, not lengths. Per-function impact contains 0.05 × effective_loc; weighting by it re-imports length through the side door. Impact stays the hitlist's ranking key.
Decisions to approve before Phase 2 starts
D1 The score is a ratio over extracted units; no per-LOC term, no sigmoid, no small-file floor.
D2public(u) is the name-set union above. This is the same set risk_api_exposure should read; that formula's own contract is a separate epic but shares Phases 1–2 here.
D3documented(u) is the header-anchored doc match, not docstring. docstring is left as is (it feeds the brief's text), not deleted.
D4 Ownership, doc_loc, opaque_execution, popularity/silo multipliers and mp leave the equation. path-and-extension-modifiers in the ledger drops risk_documentation from its signal list when this lands.
D5 No damping knob for one-unit files (0 or 100). Roll-ups are already mass-weighted. Revisit only if Phase 5 shows hitlist noise.
D6 Engine keeps emitting a float; n/a is inferred by the reporting layer from unit count 0, the same convention the corpus tooling already uses for rule absence (no None in the risk vector).
Expected values (acceptance numbers)
file
today
after
rosetta a/b/c, every language
17.9 → 79.2 depending on api assembly
100 (3 public units, 0 documented)
rosetta main, entry documented
33.4
75 where the api rule sees entry, 86 where it does not (an api contract cell, surfaced on purpose)
css, yaml, html, sqlite, markdown
0 → 55
n/a
500-line file, 10 public units, 2 documented
≈ 15
80
20-line file, 3 public units, 0 documented
40.6
100
length-leak row (bias_report.md)
rho −0.83, leak
rho 0 by construction
Phases
0 · Design freezegitgalaxy — docs/risk_documentation_contract.md (the contract sentence, the equation, the leave-list table, D1–D6), and the per-unit decomposition that produced the cluster table above committed as tests/tools/audit_documentation_inputs.py (scan → per-file arch_api, credit, doc_loc, score; reproduces every corpus score from the inputs). Approve D1–D6 on this issue. done when the contract doc is merged as draft and the acceptance table above is agreed.
1 · The name set is right before anything reads itgitgalaxy + keyword-rosetta — prerequisites, all already filed or owed:
2 · Per-unit attributes in the detectorgitgalaxy (T2) — three attributes on every extracted unit: is_public (name-set union, D2), is_documented (header-window doc match, D3), reflection_hits (the unit's own reflection_metaprogramming count, already sliced per function). Recorded as three function_data columns and in the audit JSON. Counts do not move: golden masters change by new columns only.
contract tests in tests/core_engine/test_detector.py: one per declaration family (docstring-position, preceding-block, export-list, positional-invocation languages)
keyword-rosetta: verify_language.py reads function_data; manifests gain units_public / units_documented per file; the rule-contract-audit skill run once across 46 languages for the two attributes done when 46/46 languages read public = 3, documented = 0 in a/b/c; main reads documented = 1 (entry) and public 3 or 4 with the split ledgered against the api contract; no unexplained cell.
3 · The equationgitgalaxy (T2, label rosetta:rebless-owed) — replace _calc_documentation with the ratio; delete the six tuning keys in analysis_lens.py, add public_weight and umbrella_shield; call site in signal_processor.py passes units + umbrella only.
tests: tests/core_engine/test_signal_processor.py (new pins), test_uef_length_invariance.py and tests/tools/audit_length_invariance.py (documentation is now exactly length-invariant: assert equality, not tolerance), tests/tools/audit_risk_equations.py (no tier/fid term left to audit for this formula)
golden masters: both legs re-blessed via tests/tools/crucible_check.py --update --yes; every file's documentation score moves, so the diff is scoped by the decomposition tool before blessing
prose that prints the equation: llm_recorder.py ("12. Documentation Risk Exposure …"), docs/wiki/08-03-transforming-regex-counts.md (the dynamism row), docs/gitgalaxy_architecture_brief.md self-scan regen done when both golden legs are blessed with a scoped diff, the length audit reads rho 0, and the corpus reads the acceptance table.
4 · Corpus re-bless and scoreboardkeyword-rosetta — manifests to the acceptance values; ledger: path-and-extension-modifiers drops risk_documentation, m4-makefile-api-scores-above-stratum and the api-declared-orphans-blind-to-non-word-names documentation half retire on Phase 1's verifying scan; bias_report.py: risk_dependencies() learns the per-unit inputs so a documentation cell attributes to a unit, reads_constant flips to False (the formula stops reading fid, so the metric is no longer banded within a strictness stratum, which today hides yacc at +28% against the global median), the leak table row becomes the proof. done whenrisk_documentation is 46/46 in band (n/a where declared), 0 open, and the report bands it against the global median.
5 · Real-code validationgitgalaxy — before/after on the language-crucible and on the two README example repos (curl, cics-genapp): score distribution, top-20 documentation hitlist diff, count of one-unit files at 100 and whether any reaches a hitlist. Decide D5 on evidence. docs/incidence_report.md in keyword-rosetta gains the documentation row. done when the before/after comment is on this epic and D5 is confirmed or a damping design is filed as its own issue.
6 · Closure — the contract row goes stated (score contracts get their own section in signal_contracts.py / docs/signal_contracts.md, the first entry); docs/contract_roadmap.md §4 Phase 4 marks this formula done; README "Accuracy, measured" copy refreshed with the live numbers (the paragraph still carries 2026-09-06's 90% / 2.7%; docs/validation.md carries 75% / 3 of 33). done when open-defect cells attributable to risk_documentation = 0, the sheet row is stated, and the next score contract (risk_api_exposure) can copy this epic's phase list.
Sequencing
0 design freeze ──► 1 name set (4 items, independent of each other)
│
▼
2 per-unit attributes (counts unchanged)
│
▼
3 equation + golden bless ──► 4 corpus re-bless ──► 5 real-code check ──► 6 close
Phase 1 and Phase 2 can run in parallel; Phase 3 must not start until both are merged, because the ratio reads the name set and a wrong set would be blessed into the golden masters. One layer per engine PR (rule | slicer | scoring), as #2812 rule 4.
Non-goals
risk_api_exposure's own formula (its contract is the next epic; it inherits Phases 1–2 unchanged).
Changing impact, docstring extraction, or the doc umbrella's computation.
Any damping for small files unless Phase 5 shows the need (D5).
Child of #2812 (Phase 4 · score contracts). First formula to get a score contract; the pattern here is meant to be reused for
risk_api_exposure,risk_tech_debtandfunc_complexity_gini, the other score-layer rows still carrying open-defect cells.Labels:
epicmetricscore-engineroadmapWhy this epic exists (one paragraph)
_calc_documentationdoes not measure documentation. On the rosetta corpus every term but one is flat: no function has impact above 50 (the "opaque execution" term is dead), every file is under the 50-line evidence floor (the denominator is a constant 70), and the multipliers do not move the observed values. What remains is a sigmoid over2 × api, so one adjusted-api hit per file is worth about 13 points on a 0–100 scale. The adjusted api count is assembled from two overlapping sources (the language's api rule plus the orphan-census credit, minus a tokenizer-dependent overlap), and every way that assembly fails lands in a different score cluster: 79 where a name is counted twice (m4, makefile, scheme, cobol), 55 where the rule sees one token and the census credits three (css, agc_assembly, yacc), 18 where the census is unanswerable or broken (dockerfile, html, yaml, haskell), 0 where the rule matches nothing (jcl, sqlite, objective-c main). On real code the same equation flips regime: a 500-line file with the same three undocumented public functions scores about 15, because the denominator is now the file length. That is the −0.83 length leak the bias report already flags, seen from the other side. Four of the eleven open-defect cells on the scoreboard and the corpus's single confirmed length leak are this one formula.The goal restated:
risk_documentationis the share of a file's callable surface a reader cannot recover from documentation. A ratio over units, not a density over lines. Identical intent scores identically by construction, not by tuning.The contract
Definitions that carry the contract:
apirule matched on the unit's own header ∪ names in an export list (_visibility_export/_visibility_export_list, the unreferenced_by_name reads 0.25/file for haskell and scheme: a module export list and a type signature are declarations, not references #2823 machinery) ∪ units left uncalled in a file that something imports (the Contextual Baseline Fix's orphans). A union of names: a unit cannot be public twice, and a tokenizer miss costs one unit rather than doubling the file. This is the per-unit form of raw_arch_api bands a per-file declaration count against a per-function median — one granularity mismatch, three red columns #2771's complaint (a per-file declaration count banded against a per-function median).doc-rule match whose span ends inside the unit's header window: the k lines immediately beforestart_line, or the leading lines of the body (docstring position). Not the slicer'sdocstringfield, which is a per-language "preceding comment" heuristic: on the corpus it captures python's planted# HACK:comment as a docstring, reads 13 of 13 for jcl, and 0 for fortran, haskell and scheme.doc_umbrellaas today (GuideStar folder coverage, README-upgraded). The only file-level defence that survives.What leaves the equation, with the reason:
doc_loc × 0.33ownership × 0.5opaque_execution(impact > 50 ∧ no docstring)/ (_mass_loc(loc) + 20)and the sigmoidmpdoc_weight,ownership_weight,doc_loc_weight,loc_smoothing,threshold_base,sigmoid_slopepublic_weight(2.0),umbrella_shield(0.5)Weights are counts, not lengths. Per-function
impactcontains0.05 × effective_loc; weighting by it re-imports length through the side door. Impact stays the hitlist's ranking key.Decisions to approve before Phase 2 starts
public(u)is the name-set union above. This is the same setrisk_api_exposureshould read; that formula's own contract is a separate epic but shares Phases 1–2 here.documented(u)is the header-anchoreddocmatch, notdocstring.docstringis left as is (it feeds the brief's text), not deleted.mpleave the equation.path-and-extension-modifiersin the ledger dropsrisk_documentationfrom its signal list when this lands.n/ais inferred by the reporting layer from unit count 0, the same convention the corpus tooling already uses for rule absence (noNonein the risk vector).Expected values (acceptance numbers)
entrydocumentedentry, 86 where it does not (anapicontract cell, surfaced on purpose)bias_report.md)Phases
0 · Design freeze
gitgalaxy—docs/risk_documentation_contract.md(the contract sentence, the equation, the leave-list table, D1–D6), and the per-unit decomposition that produced the cluster table above committed astests/tools/audit_documentation_inputs.py(scan → per-filearch_api, credit,doc_loc, score; reproduces every corpus score from the inputs). Approve D1–D6 on this issue.done when the contract doc is merged as
draftand the acceptance table above is agreed.1 · The name set is right before anything reads it
gitgalaxy + keyword-rosetta— prerequisites, all already filed or owed:\b\w+\b, so a hyphenated or qualified name is never recognised as already-declared (#2731's double count returns for cobol and scheme) #2827 —api_declared_orphanstokenizer (\b\w+\bcannot seePROBE-GLOBALS,probe-globals); cobol + scheme, 3 cellsapirule counts top-level variable declarations (int shared_region =indata/c/a.creads api 5 for 3 functions);raw_arch_api3.75 vs median 3 sits at exactly +25.0%, in band by zero margin, unledgereddone when
arch_apireads 3 in every a/b/c file that has an api rule, and therisk_api_exposure/raw_arch_api/risk_documentationopen cells attributable to these are 0.2 · Per-unit attributes in the detector
gitgalaxy(T2) — three attributes on every extracted unit:is_public(name-set union, D2),is_documented(header-windowdocmatch, D3),reflection_hits(the unit's ownreflection_metaprogrammingcount, already sliced per function). Recorded as threefunction_datacolumns and in the audit JSON. Counts do not move: golden masters change by new columns only._public_name_set(file)from api header matches +_export_declaration_offsets+ healed orphans;_doc_anchored(unit)over the comment stream (thedocrule reads comments, per ownership has no stated contract: solidity counts the SPDX license line on every file, five languages count copyright notices and license prose, three count the phrase 'created by' in sentences, and five doc rules still double-count the author tag #2882's stream convention)tests/core_engine/test_detector.py: one per declaration family (docstring-position, preceding-block, export-list, positional-invocation languages)keyword-rosetta:verify_language.pyreadsfunction_data; manifests gainunits_public/units_documentedper file; therule-contract-auditskill run once across 46 languages for the two attributesdone when 46/46 languages read public = 3, documented = 0 in a/b/c; main reads documented = 1 (
entry) and public 3 or 4 with the split ledgered against theapicontract; no unexplained cell.3 · The equation
gitgalaxy(T2, labelrosetta:rebless-owed) — replace_calc_documentationwith the ratio; delete the six tuning keys inanalysis_lens.py, addpublic_weightandumbrella_shield; call site insignal_processor.pypasses units + umbrella only.tests/core_engine/test_signal_processor.py(new pins),test_uef_length_invariance.pyandtests/tools/audit_length_invariance.py(documentation is now exactly length-invariant: assert equality, not tolerance),tests/tools/audit_risk_equations.py(no tier/fid term left to audit for this formula)tests/tools/crucible_check.py --update --yes; every file's documentation score moves, so the diff is scoped by the decomposition tool before blessingllm_recorder.py("12. Documentation Risk Exposure …"),docs/wiki/08-03-transforming-regex-counts.md(the dynamism row),docs/gitgalaxy_architecture_brief.mdself-scan regendone when both golden legs are blessed with a scoped diff, the length audit reads rho 0, and the corpus reads the acceptance table.
4 · Corpus re-bless and scoreboard
keyword-rosetta— manifests to the acceptance values; ledger:path-and-extension-modifiersdropsrisk_documentation,m4-makefile-api-scores-above-stratumand theapi-declared-orphans-blind-to-non-word-namesdocumentation half retire on Phase 1's verifying scan;bias_report.py:risk_dependencies()learns the per-unit inputs so a documentation cell attributes to a unit,reads_constantflips to False (the formula stops readingfid, so the metric is no longer banded within a strictness stratum, which today hides yacc at +28% against the global median), the leak table row becomes the proof.done when
risk_documentationis 46/46 in band (n/a where declared), 0 open, and the report bands it against the global median.5 · Real-code validation
gitgalaxy— before/after on the language-crucible and on the two README example repos (curl, cics-genapp): score distribution, top-20 documentation hitlist diff, count of one-unit files at 100 and whether any reaches a hitlist. Decide D5 on evidence.docs/incidence_report.mdin keyword-rosetta gains the documentation row.done when the before/after comment is on this epic and D5 is confirmed or a damping design is filed as its own issue.
6 · Closure — the contract row goes
stated(score contracts get their own section insignal_contracts.py/docs/signal_contracts.md, the first entry);docs/contract_roadmap.md§4 Phase 4 marks this formula done; README "Accuracy, measured" copy refreshed with the live numbers (the paragraph still carries 2026-09-06's 90% / 2.7%;docs/validation.mdcarries 75% / 3 of 33).done when open-defect cells attributable to
risk_documentation= 0, the sheet row isstated, and the next score contract (risk_api_exposure) can copy this epic's phase list.Sequencing
Phase 1 and Phase 2 can run in parallel; Phase 3 must not start until both are merged, because the ratio reads the name set and a wrong set would be blessed into the golden masters. One layer per engine PR (rule | slicer | scoring), as #2812 rule 4.
Non-goals
risk_api_exposure's own formula (its contract is the next epic; it inherits Phases 1–2 unchanged).impact,docstringextraction, or the doc umbrella's computation._calc_*; the commensurability audit tool in Epic: contracts at every layer — stream, count, score (the rosetta roadmap; child of #2560) #2812 Phase 4 is a separate deliverable this epic will feed one worked example into.Issue disposition
raw_arch_apigranularity)irc)risk_documentationrisk-documentation-zero-evidence-guard(retired)n/a(D6); note on the entry, no revivalEvidence behind the "why"
Sensitivity of today's equation to the adjusted api count per file (all other inputs at the corpus values, doc_loc = 2):
risk_documentationWhere the corpus lands and why (per-file scan of 20 languages against engine main, 2026-09-09; every score reproduced from the formula):