[jdkent]

What changes

A record is filled in three passes: values, evidence, then normalization. This gives the third pass one place to write, on every extracted value: ExtractedValue.normalized, a list of Normalization entries.

Slot	Meaning
vocabulary (required)	answer set: MONDO, ONVOC, AgeUnit, a canonical task set
status (required)	mapped, unmapped (read, could not place), ambiguous, absent (value states an absence)
value, label	the term
item, head	which list item, and which part of a comorbidity string, the entry reads
qualifiers, rollup	words lifted off before lookup; whether the term is broader than the source
method, also_read	how the term was reached; other slots read (entity-level normalizations sit on name)
mapping_tool (required), confidence	SSSOM's names: the normalizer and version, and its score

• Migrated: Group.age_unit_normalized and CategoryDistribution.category_normalized leave extraction (deviations deleted, tree regenerated with pondie.schema.generate). Storage keeps both deterministic columns, now mapped from the sidecar.
• Not normalizations: is_healthy and population_characteristics stay derived slots; cross-entity relations stay in derived tables.
• Versions: extraction-evidence 0.3.0, extraction-to-storage map 0.8.0. extraction-readme.md documents the pass and narrows the immutability rule to "normalization writes only normalized".

Why

• Each sibling cost a storage slot, a deviation and a map entry, and was written as an ExtractedValue with evidence: not_found.
• The other normalizers (MONDO/ONVOC conditions, canonical tasks, medication status, modality) wrote only to tables outside the records.
• A sibling cannot hold per-item or multi-vocabulary readings, which medical_condition needs.

Checks

• pondie.schema.generate --check, and pondie's extraction_to_storage_map, linkml, field_provenance and storage_parameter_priorities checks, all pass against this branch (PONDIE_SCHEMA_DIR).

• tools/generate_models.py --check and the full test suite pass. The new tests/test_normalized.py:

  • every wrapper carries normalized;
  • both the model and the JSON Schema accept a record with sidecars;
  • both refuse an unknown status, confidence > 1, a negative item, and a missing vocabulary or mapping_tool;
  • the siblings are gone from extraction and kept in storage.

  These tests fail on main.

• Prototype on two real out_all2 records (14 sidecars across age unit, sex category, MONDO+ONVOC on medical_condition, canonical task on Task.name): 0 new LinkML errors; the previous schema rejects every sidecar.

Follow-up (pondie, not in this PR)

• age_unit, sex_distribution and handedness_distribution must write sidecars instead of siblings. Records written by the current pondie fail this schema until then, because the extraction models refuse undeclared fields.
• Field modules should emit their decisions as sidecars.
• Migrate existing records.
• Make the astra build read groups_medical and tasks from the records.

🤖 Generated with Claude Code

[jdkent]

LGTM