Skip to content

Epic: risk_documentation becomes a per-unit coverage ratio (Phase 4 score contract; child of #2812) #2908

Description

@squid-protocol

Child of #2812 (Phase 4 · score contracts). First formula to get a score contract; the pattern here is meant to be reused for risk_api_exposure, risk_tech_debt and func_complexity_gini, the other score-layer rows still carrying open-defect cells.

Labels: epic metrics core-engine roadmap

Why this epic exists (one paragraph)

_calc_documentation does not measure documentation. On the rosetta corpus every term but one is flat: no function has impact above 50 (the "opaque execution" term is dead), every file is under the 50-line evidence floor (the denominator is a constant 70), and the multipliers do not move the observed values. What remains is a sigmoid over 2 × api, so one adjusted-api hit per file is worth about 13 points on a 0–100 scale. The adjusted api count is assembled from two overlapping sources (the language's api rule plus the orphan-census credit, minus a tokenizer-dependent overlap), and every way that assembly fails lands in a different score cluster: 79 where a name is counted twice (m4, makefile, scheme, cobol), 55 where the rule sees one token and the census credits three (css, agc_assembly, yacc), 18 where the census is unanswerable or broken (dockerfile, html, yaml, haskell), 0 where the rule matches nothing (jcl, sqlite, objective-c main). On real code the same equation flips regime: a 500-line file with the same three undocumented public functions scores about 15, because the denominator is now the file length. That is the −0.83 length leak the bias report already flags, seen from the other side. Four of the eleven open-defect cells on the scoreboard and the corpus's single confirmed length leak are this one formula.

The goal restated: risk_documentation is the share of a file's callable surface a reader cannot recover from documentation. A ratio over units, not a density over lines. Identical intent scores identically by construction, not by tuning.

The contract

risk_documentation — of the units extracted from a file (functions, methods, paragraphs, steps), the weight-share that is undocumented. A unit is public when its name is in the file's public name set. A unit is documented when a doc-rule match is anchored to its header window. Public units weigh double; a unit's weight rises by one per reflection_metaprogramming hit inside it. A folder-level documentation umbrella shields the whole file multiplicatively. A file with no extracted units has no value (n/a), not zero.

for each extracted unit u:
    weight(u)  = (2 if public(u) else 1) + reflection_hits_inside(u)
    exposed(u) = weight(u) if not documented(u) else 0

risk_documentation = 100 × Σ exposed(u) / Σ weight(u) × (1 − umbrella_shield × doc_umbrella)
                   = n/a when Σ weight(u) = 0

Definitions that carry the contract:

What leaves the equation, with the reason:

term why it goes
doc_loc × 0.33 a comment-line count is not coverage, and it is one of the two length terms
ownership × 0.5 who maintains a file is a different question, already carried by silo/authorship views
opaque_execution (impact > 50 ∧ no docstring) dead on small functions; its intent (load-bearing undocumented code) is what public weighting expresses
/ (_mass_loc(loc) + 20) and the sigmoid the length regime; the ratio is already 0–100
popularity and silo multipliers, path modifier mp blast radius sits next to the score in the same report row; a coverage ratio does not change because the file lives in a config folder
doc_weight, ownership_weight, doc_loc_weight, loc_smoothing, threshold_base, sigmoid_slope six knobs become two: public_weight (2.0), umbrella_shield (0.5)

Weights are counts, not lengths. Per-function impact contains 0.05 × effective_loc; weighting by it re-imports length through the side door. Impact stays the hitlist's ranking key.

Decisions to approve before Phase 2 starts

  • D1 The score is a ratio over extracted units; no per-LOC term, no sigmoid, no small-file floor.
  • D2 public(u) is the name-set union above. This is the same set risk_api_exposure should read; that formula's own contract is a separate epic but shares Phases 1–2 here.
  • D3 documented(u) is the header-anchored doc match, not docstring. docstring is left as is (it feeds the brief's text), not deleted.
  • D4 Ownership, doc_loc, opaque_execution, popularity/silo multipliers and mp leave the equation. path-and-extension-modifiers in the ledger drops risk_documentation from its signal list when this lands.
  • D5 No damping knob for one-unit files (0 or 100). Roll-ups are already mass-weighted. Revisit only if Phase 5 shows hitlist noise.
  • D6 Engine keeps emitting a float; n/a is inferred by the reporting layer from unit count 0, the same convention the corpus tooling already uses for rule absence (no None in the risk vector).

Expected values (acceptance numbers)

file today after
rosetta a/b/c, every language 17.9 → 79.2 depending on api assembly 100 (3 public units, 0 documented)
rosetta main, entry documented 33.4 75 where the api rule sees entry, 86 where it does not (an api contract cell, surfaced on purpose)
css, yaml, html, sqlite, markdown 0 → 55 n/a
500-line file, 10 public units, 2 documented ≈ 15 80
20-line file, 3 public units, 0 documented 40.6 100
length-leak row (bias_report.md) rho −0.83, leak rho 0 by construction

Phases

  • 0 · Design freeze gitgalaxydocs/risk_documentation_contract.md (the contract sentence, the equation, the leave-list table, D1–D6), and the per-unit decomposition that produced the cluster table above committed as tests/tools/audit_documentation_inputs.py (scan → per-file arch_api, credit, doc_loc, score; reproduces every corpus score from the inputs). Approve D1–D6 on this issue.
    done when the contract doc is merged as draft and the acceptance table above is agreed.

  • 1 · The name set is right before anything reads it gitgalaxy + keyword-rosetta — prerequisites, all already filed or owed:

  • 2 · Per-unit attributes in the detector gitgalaxy (T2) — three attributes on every extracted unit: is_public (name-set union, D2), is_documented (header-window doc match, D3), reflection_hits (the unit's own reflection_metaprogramming count, already sliced per function). Recorded as three function_data columns and in the audit JSON. Counts do not move: golden masters change by new columns only.

  • 3 · The equation gitgalaxy (T2, label rosetta:rebless-owed) — replace _calc_documentation with the ratio; delete the six tuning keys in analysis_lens.py, add public_weight and umbrella_shield; call site in signal_processor.py passes units + umbrella only.

    • tests: tests/core_engine/test_signal_processor.py (new pins), test_uef_length_invariance.py and tests/tools/audit_length_invariance.py (documentation is now exactly length-invariant: assert equality, not tolerance), tests/tools/audit_risk_equations.py (no tier/fid term left to audit for this formula)
    • golden masters: both legs re-blessed via tests/tools/crucible_check.py --update --yes; every file's documentation score moves, so the diff is scoped by the decomposition tool before blessing
    • prose that prints the equation: llm_recorder.py ("12. Documentation Risk Exposure …"), docs/wiki/08-03-transforming-regex-counts.md (the dynamism row), docs/gitgalaxy_architecture_brief.md self-scan regen
      done when both golden legs are blessed with a scoped diff, the length audit reads rho 0, and the corpus reads the acceptance table.
  • 4 · Corpus re-bless and scoreboard keyword-rosetta — manifests to the acceptance values; ledger: path-and-extension-modifiers drops risk_documentation, m4-makefile-api-scores-above-stratum and the api-declared-orphans-blind-to-non-word-names documentation half retire on Phase 1's verifying scan; bias_report.py: risk_dependencies() learns the per-unit inputs so a documentation cell attributes to a unit, reads_constant flips to False (the formula stops reading fid, so the metric is no longer banded within a strictness stratum, which today hides yacc at +28% against the global median), the leak table row becomes the proof.
    done when risk_documentation is 46/46 in band (n/a where declared), 0 open, and the report bands it against the global median.

  • 5 · Real-code validation gitgalaxy — before/after on the language-crucible and on the two README example repos (curl, cics-genapp): score distribution, top-20 documentation hitlist diff, count of one-unit files at 100 and whether any reaches a hitlist. Decide D5 on evidence. docs/incidence_report.md in keyword-rosetta gains the documentation row.
    done when the before/after comment is on this epic and D5 is confirmed or a damping design is filed as its own issue.

  • 6 · Closure — the contract row goes stated (score contracts get their own section in signal_contracts.py / docs/signal_contracts.md, the first entry); docs/contract_roadmap.md §4 Phase 4 marks this formula done; README "Accuracy, measured" copy refreshed with the live numbers (the paragraph still carries 2026-09-06's 90% / 2.7%; docs/validation.md carries 75% / 3 of 33).
    done when open-defect cells attributable to risk_documentation = 0, the sheet row is stated, and the next score contract (risk_api_exposure) can copy this epic's phase list.

Sequencing

0 design freeze ──► 1 name set (4 items, independent of each other)
                         │
                         ▼
                    2 per-unit attributes (counts unchanged)
                         │
                         ▼
                    3 equation + golden bless ──► 4 corpus re-bless ──► 5 real-code check ──► 6 close

Phase 1 and Phase 2 can run in parallel; Phase 3 must not start until both are merged, because the ratio reads the name set and a wrong set would be blessed into the golden masters. One layer per engine PR (rule | slicer | scoring), as #2812 rule 4.

Non-goals

Issue disposition

issue action
#2827, #2871 Phase 1 prerequisites; keep, unchanged
#2872 closed (engine half #2902); the corpus re-plant is Phase 1 item 2
#2771 (raw_arch_api granularity) answered by D2's per-unit name set; close when Phase 2 lands
#2719 (dynamism replaces flat irc) kept, moved per unit (D1); no reversal
#2655 (evidence-mass floor) the floor stays for the other density formulas; this formula stops reading it
bias report length-leak row for risk_documentation resolved by construction in Phase 3; the row stays as the proof
keyword-rosetta risk-documentation-zero-evidence-guard (retired) the guard's role (no surface → 0) becomes n/a (D6); note on the entry, no revival

Evidence behind the "why"

Sensitivity of today's equation to the adjusted api count per file (all other inputs at the corpus values, doc_loc = 2):

adjusted api / file 0 1 2 3 4 5 6 7
risk_documentation 0 17.9 27.9 40.6 54.8 68.2 79.2 86

Where the corpus lands and why (per-file scan of 20 languages against engine main, 2026-09-09; every score reproduced from the formula):

adjusted api / file languages (a/b/c) mechanism
6 makefile, m4, scheme, cobol raw 3 + orphan credit 3: overlap check never recognises the names (#2827, #2872)
5 c (a.c only) globals plants match the api rule as top-level declarations
4 css, agc_assembly, yacc raw 1 + credit 3; yacc adds 4 reflection hits
3 the mainstream pack; objective-c via credit alone (raw 0) intended
1 dockerfile, html, yaml (census n/a), haskell (#2871) raw 1, no credit
0 jcl, sqlite, objective-c main zero-evidence guard

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    core-engineModifications to the central physics and parsing engineepicLarge-scale architectural milestone or multi-issue initiativemetricsHeuristics, risk exposures, and topological math updatesroadmapLong-term strategic planning and upcoming feature tracking

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions