[derek73]

Extends the native-script Wikidata corpus (#693) to South Asian scripts it held little or nothing of.

What changed

Four new strata, and the Hindi one grown, in build_wikidata_native_corpus.py. Yields were measured 2026-10-23 with no quota cap. The India column is in build order, because the four India strata share their people: each takes no one an earlier one took.

Stratum	Quota / floor	Pool	Yield in build order	Yield alone
hi-in	15 → 60 / 50	1,000 → 5,000, whole	147 of 4,806	149
mr-in	30 / 25	3,000	40 of 3,000	50
ur-in	12 / 10	3,000	12 of 3,000	17
ur-pk	20 / 15	1,600, whole	24 of 1,456	24
ne-np	10 / 8	200, whole	12 of 170	12

ur-in yields exactly its quota in build order, so its floor of 10 is its only margin. If a later re-pool loses three of its people, the build stops there.

Dari in Afghanistan (fa-af) was measured and left out: 4 of 291 aligned.

The yields are low for the reason the builder already documents: these people's given-name and family-name items are mostly labelled in Latin script only, so the strict word-by-word alignment refuses them. Larger pools were the way to get usable strata without loosening that rule, for example by matching through transliteration.

hi-in stays in its old position ahead of mr-in and ur-in. All 15 of its existing names stay in, because the MD5 order makes the old 1,000-person pool a prefix of the new 5,000.

Effect

• 910 → 1,027 renderings. No existing row moved, and all 117 new names are new to snapshot.jsonl (117 lines added, 0 removed).
• wikidata_native_agreement.json gains two names, four role disagreements: the Urdu two-word surnames رضا خان and اگنی ہوتری, where the parser reads the first word as a middle name. That's the same class as the Arabic عبد العزيز already recorded.
• All 85 new Devanagari names carry combining vowel signs, and all agree. This says nothing about the gap rules/H2's Accepted clause records, a period-marked abbreviation in an abugida, because no text in the corpus holds a period. The 32 Urdu names carry no combining mark at all.
• One known data error is kept as-is, since the builder filters nothing by hand. Wikidata states Rahul Bajaj (राहुल बजाज, mr-in) with his given-name and family-name items swapped, so the row is scored family-first and agrees.

Docs

The builder docstring and tools/differential/README.md (roster row and native-source paragraph) now name the new strata and their yields. Both used to say Urdu and Marathi were left out.

Testing

• uv run pytest: 23,332 passed (the serial benchmark tests were not run; they don't read the corpora). After the review fix, the corpus, agreement and snapshot test modules pass again (161 passed).
• uv run ruff check, uv run mypy: clean.
• cldr_agreement.py --record and snapshot.py --record re-run. The JA locale rows were not measured locally (namedivider absent), and their recorded entries are kept unchanged.
• Review: an in-memory rebuild from the cache, with the network disabled, reproduces the committed fields file row for row.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.05%. Comparing base (204d9f4) to head (6fb0a5c).

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #862   +/-   ##
=======================================
  Coverage   99.05%   99.05%           
=======================================
  Files          47       47           
  Lines        4669     4669           
=======================================
  Hits         4625     4625           
  Misses         44       44           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.