Add Hindi, Marathi, Urdu and Nepali strata to the native-script corpus

tools/differential/README.md

@@ -199,7 +199,7 @@ invocation shows it.
 | `corpus_cldr.jsonl` | Watch, Score | Unicode CLDR's person-name formatting test data at `release-48`: each fictional name's long, formal rendering in givenFirst, surnameFirst and sorting order, about 110 locales in native scripts, via `build_cldr_corpus.py` (2026-10-12); Unicode-3.0, notice in `cldr_LICENSE` | real-world name distributions: the names are written to fill every field, so full-house names are over-represented and plain two-word names under-represented |
 | `corpus_gnd.jsonl` | Watch, Score | the German National Library's authority file (GND, via lobid), real people who are deceased and have a Wikipedia article: each preferred name as cataloged (`Gogh, Vincent van`) and, outside family-first areas, a given-first rendering composed from its fields (a regnal name is composed only, `Johann II.`, without its German descriptor), sampled by shape and area via `build_gnd_corpus.py` (2026-10-16); CC0, credit in `gnd_LICENSE` | names as their bearers write them: German library romanization and German forms for historic figures, Latin script only, titles, credentials and nicknames kept out of the preferred name (a few slip into a `personalName`: `Imam Bonjol`), and the catalog's entry rules (not the bearer's culture) decide what counts as a surname |
 | `corpus_wikidata.jsonl` | Watch, Score | Wikidata, real public figures (deceased, or with at least 10 sitelinks) whose item states a second surname (P1950: Spain, Mexico, Colombia) or a patronymic (P5056: Russia, the Soviet Union, the Russian Empire, Iceland, Malaysia): each P1559 native-language name and own-language label whose every word aligns with exactly one stated part, via `build_wikidata_corpus.py` (2026-10-16); CC0, credit in `wikidata_LICENSE` | everything the strict alignment drops: short display forms without the second surname, nicknames, and names with an unrecorded part, so tidy names are over-represented; only those eight citizenships; no titles, credentials or suffixes |
-| `corpus_wikidata_native.jsonl` | Watch, Score | Wikidata, real public figures (deceased, or with at least 10 sitelinks) with a name written in their own script: Hangul, Han and kana, Arabic (Syria, Egypt, Saudi Arabia, Iran), Hebrew, Georgian, Armenian, Greek, Cyrillic beyond Russian (Serbia, Bulgaria, Ukraine, Kazakhstan), Bengali, Devanagari and Thai, 20 strata sampled by country and language. Each person's first P1559 or own-language label whose every word, or every character in unspaced text, aligns with exactly one stated part, via `build_wikidata_native_corpus.py` (2026-10-16). CC0, credit in `wikidata_native_LICENSE` | everything the strict alignment drops, which in these scripts is most names: Arabic nasab chains (`بن` is no connective in Arabic script), `آل` family names and kunyas (most Arabic items state no family name), Chinese names whose given name has no item (most), Japanese names whose given-name item holds only a kana reading, and names whose parts are labelled only in Latin script (most Devanagari, Bengali and Thai items, so those strata are 10–25 names); transcriptions with the 间隔号; Urdu and Marathi entirely; no titles, credentials or suffixes |
+| `corpus_wikidata_native.jsonl` | Watch, Score | Wikidata, real public figures (deceased, or with at least 10 sitelinks) with a name written in their own script: Hangul, Han and kana, Arabic (Syria, Egypt, Saudi Arabia, Iran, and Urdu in Pakistan and India), Hebrew, Georgian, Armenian, Greek, Cyrillic beyond Russian (Serbia, Bulgaria, Ukraine, Kazakhstan), Bengali, Devanagari (Hindi, Marathi, Nepali) and Thai, 24 strata sampled by country and language. Each person's first P1559 or own-language label whose every word, or every character in unspaced text, aligns with exactly one stated part, via `build_wikidata_native_corpus.py` (2026-10-16; the Hindi stratum grown and the Marathi, Urdu and Nepali strata added 2026-10-23). CC0, credit in `wikidata_native_LICENSE` | everything the strict alignment drops, which in these scripts is most names: Arabic nasab chains (`بن` is no connective in Arabic script), `آل` family names and kunyas (most Arabic items state no family name), Chinese names whose given name has no item (most), Japanese names whose given-name item holds only a kana reading, and names whose parts are labelled only in Latin script (most Devanagari, Bengali, Urdu and Thai items, so those strata are 10–60 names: Hindi's whole population of 4,806 yields 147 in build order, Indian Urdu 12 of 3,000, on 2026-10-23); transcriptions with the 间隔号; Dari entirely; no titles, credentials or suffixes |
 | `corpus_conventions.jsonl` | Watch, Score | invented names patterned on naming conventions the other corpora barely cover -- no family name, religious names, positional patronymics, honorifics after the name -- from the hand-written table in `conventions.py`, via `build_conventions_corpus.py` (#686) | real names, and every convention nobody has written a row for: about one row per convention, so it shows that a convention moved, not how often |
 | `corpus_rules.jsonl` | Watch | every example in `docs/design/rules/`, via `build_rules_corpus.py` (#414) | anything the rules doc has no example for |
 | `corpus_shapes.jsonl` | Watch | shape-tagged rows of `tests/v2/cases.py`, via `build_shapes_corpus.py` (#468) | anything no one has tagged a row for |
@@ -482,7 +482,7 @@ A second view measures the locale packs (#672). `LOCALE_FILES` in `cldr_agreemen
 
 **The Wikidata source.** `build_wikidata_corpus.py` exists for the two fields no other open source labels, and states them in `wikidata_fields.jsonl` as `surname2` (P1950, CLDR's own field name) and `patronymic` (P5056, which CLDR has no field for), beside `given` (P735) and `surname` (P734); its disagreements are in `wikidata_agreement.json`. A person's written names are token-aligned to the parts rather than composed from them: a text is kept only where every word is claimed by exactly one part, a connective (`bin`, `i`, `y`) joining the part after it, and a woman's Russian surname may match the feminine form of its item's label. ParsedName has no `surname2` or `patronymic` attribute, so `cldr_agreement.py` lists them in `PENDING_ROLES` and scores them inside `family` and `given_names` respectively -- a patronymic is not a family name, so `Vigdís Finnbogadóttir` expects the family empty -- and also scores each on its own the day ParsedName gains an attribute of that name (whether the merged role then drops it is for the 3.0 design). Until then the summary says how many renderings state each. CLDR's `surname2` is in the same position. A stratum that falls below its floor, a query that matches nothing, a pool short of its size (unless the stratum is marked `whole`, which a full pool then contradicts) and a response that is not what was asked for each stop the build before anything is written; Iceland (270 of a pool of 600 on 2026-10-16, measured in the native paragraph below) and Malaysia are marked `whole`; Malaysia's floor is set below its quota because only 17 Malaysian public figures with a P5056 statement matched the query on 2026-10-16, and 9 of them aligned. Raw responses are cached outside the tree, in a `wikidata/` directory under `$NAMEPARSER_TOOLS_CACHE` (default `~/.cache/nameparser-tools`); `--refresh` fetches them again.
 
-**The native-script Wikidata source.** `build_wikidata_native_corpus.py` exists because every other real-name corpus is Latin script, apart from Wikidata's Russian patronymics, so the script-conditional readings (family-first Han, Hangul and kana; Korean surname segmentation) had been tested only against CLDR's invented names. No text holds an honorific, so the corpus does not exercise the CJK honorific peel; it shows only that the peel leaves plain names alone. It reuses `build_wikidata_corpus.py`'s fetches and token alignment and adds the following. Strata are a country and a language, not a property, and the query keeps people with a label in that language. A text is used only if every letter and combining mark is in the stratum's script (`mul` labels are Latin: `Anushka Sharma` would otherwise pass as a Hindi name). Unspaced text is aligned by character under the same exactly-one-part rule, and nothing is inferred: a Chinese item stating only its surname is left out rather than reading the rest of the text as the given name. Feminine surname forms are kept per language (Greek `Λελούδα` for `Λελούδας`, Bulgarian and Kazakh `-ова`, Ukrainian `-ська`), so the filter does not drop women. A Han, kana or Hangul text carries no shape, since the parser reads those scripts family-first by itself and that is what is being tested; pure katakana keeps the order it is written in, as the parser does. Each person gives one rendering, P1559 first, and its disagreements are in `wikidata_native_agreement.json`. On 2026-10-16 every spaced stratum agreed in every role except seven Arabic surnames of two words (`عبد العزيز`, `عز الدين`, `شكر الله`, `ذو الغنى`: the parser reads the first word as a middle name), a Persian double surname, and a Bulgarian name whose two stated family names include what is probably the father's name (`Динко Цветков Дерменджиев`); Hangul agreed on 120 of 120; Han read by default disagrees on every unspaced Chinese name and 49 of 60 Japanese ones, while the locale view reads 74/80 under `ZH` (six surnames it does not segment: `詹`, `卫`, `柯`, `茅`, `华`, `文`) and 59/60 under `JA`. Wikidata's P734 is a convention, not a fact, where the culture has none: an Arabic nisba (`البغدادي`) is entered as a family name, and the corpus scores it as one. `--lc-check` compares each name with the native-script variants of its Library of Congress record (P244), splitting by LC's comma or, in CJK, a space, and counts an LCCN LC no longer has; it runs after the build has written its files, and adds nothing to them and filters nothing from them. A pool that comes back shorter than its size stops the build unless its stratum is marked as the whole population (Thailand, Kazakhstan, Bangladesh), so a query that starts matching less cannot drift the sample unseen; a stratum so marked whose pool comes back full stops it too, since the mark is then stale. The check is `pooled()` in `build_wikidata_corpus.py`, which both Wikidata builders call; re-pooled on 2026-10-16 (#696), the parent's Iceland returned 270 of a pool of 600 and Malaysia 17 of 240, so both are marked whole, and the other six filled their pools (es 900, mx 600, co 480, ru 660, su 360, ri 240). On 2026-10-16, 99 of the 910 names had an LC variant that splits them, and 96 agreed; the other three are Arabic names whose LC record names a person written the other way round (`Ibrāhīm, Ṣāliḥ` for Wikidata's `إبراهيم صالح`), so the P244 link or the order is suspect, not the split. LC writes most Han variants unspaced, so it confirms almost no Chinese or Japanese split. The 907 corpus lines (904 names new to the snapshot; three Ukrainian names were already in `corpus_wikidata.jsonl`) cost about 2.9 s in the corpus-reading test modules and about 0.6 s in the agreement tests, measured locally without coverage on 2026-10-16. Raw responses are cached under `$NAMEPARSER_TOOLS_CACHE`, in `wikidata/` and `lc/`.
+**The native-script Wikidata source.** `build_wikidata_native_corpus.py` exists because every other real-name corpus is Latin script, apart from Wikidata's Russian patronymics, so the script-conditional readings (family-first Han, Hangul and kana; Korean surname segmentation) had been tested only against CLDR's invented names. No text holds an honorific, so the corpus does not exercise the CJK honorific peel; it shows only that the peel leaves plain names alone. It reuses `build_wikidata_corpus.py`'s fetches and token alignment and adds the following. Strata are a country and a language, not a property, and the query keeps people with a label in that language. A text is used only if every letter and combining mark is in the stratum's script (`mul` labels are Latin: `Anushka Sharma` would otherwise pass as a Hindi name). Unspaced text is aligned by character under the same exactly-one-part rule, and nothing is inferred: a Chinese item stating only its surname is left out rather than reading the rest of the text as the given name. Feminine surname forms are kept per language (Greek `Λελούδα` for `Λελούδας`, Bulgarian and Kazakh `-ова`, Ukrainian `-ська`), so the filter does not drop women. A Han, kana or Hangul text carries no shape, since the parser reads those scripts family-first by itself and that is what is being tested; pure katakana keeps the order it is written in, as the parser does. Each person gives one rendering, P1559 first, and its disagreements are in `wikidata_native_agreement.json`. On 2026-10-16 every spaced stratum agreed in every role except seven Arabic surnames of two words (`عبد العزيز`, `عز الدين`, `شكر الله`, `ذو الغنى`: the parser reads the first word as a middle name), a Persian double surname, and a Bulgarian name whose two stated family names include what is probably the father's name (`Динко Цветков Дерменджиев`); Hangul agreed on 120 of 120; on 2026-10-23 the 117 Devanagari and Urdu names then added agreed but for two Urdu surnames of two words (`رضا خان`, `اگنی ہوتری`), the Arabic surnames' class; all 85 new Devanagari names carry combining vowel signs and agreed, though none holds a period, so the gap rules/H2's Accepted clause records (a period-marked abbreviation in an abugida) is out of the corpus's reach, and the 32 Urdu names carry no combining mark at all. One Marathi name is stated with its parts swapped (`राहुल बजाज`, given `बजाज`) and so scores as family-first; Han read by default disagrees on every unspaced Chinese name and 49 of 60 Japanese ones, while the locale view reads 74/80 under `ZH` (six surnames it does not segment: `詹`, `卫`, `柯`, `茅`, `华`, `文`) and 59/60 under `JA`. Wikidata's P734 is a convention, not a fact, where the culture has none: an Arabic nisba (`البغدادي`) is entered as a family name, and the corpus scores it as one. `--lc-check` compares each name with the native-script variants of its Library of Congress record (P244), splitting by LC's comma or, in CJK, a space, and counts an LCCN LC no longer has; it runs after the build has written its files, and adds nothing to them and filters nothing from them. A pool that comes back shorter than its size stops the build unless its stratum is marked as the whole population (Thailand, Kazakhstan, Bangladesh, Hindi in India, Urdu in Pakistan, Nepal), so a query that starts matching less cannot drift the sample unseen; a stratum so marked whose pool comes back full stops it too, since the mark is then stale. The check is `pooled()` in `build_wikidata_corpus.py`, which both Wikidata builders call; re-pooled on 2026-10-16 (#696), the parent's Iceland returned 270 of a pool of 600 and Malaysia 17 of 240, so both are marked whole, and the other six filled their pools (es 900, mx 600, co 480, ru 660, su 360, ri 240). On 2026-10-16, 99 of the 910 names had an LC variant that splits them, and 96 agreed; the other three are Arabic names whose LC record names a person written the other way round (`Ibrāhīm, Ṣāliḥ` for Wikidata's `إبراهيم صالح`), so the P244 link or the order is suspect, not the split. LC writes most Han variants unspaced, so it confirms almost no Chinese or Japanese split. The first build's 907 corpus lines (904 names new to the snapshot; three Ukrainian names were already in `corpus_wikidata.jsonl`; the 2026-10-23 strata added 117 lines, all new to it) cost about 2.9 s in the corpus-reading test modules and about 0.6 s in the agreement tests, measured locally without coverage on 2026-10-16. Raw responses are cached under `$NAMEPARSER_TOOLS_CACHE`, in `wikidata/` and `lc/`.
 
 **The conventions source.** `conventions_fields.jsonl` states the parts of the hand-written conventions corpus, from the `fields` of `conventions.py`'s rows, and its disagreements are in `conventions_agreement.json`. The names are invented, so the record is a worklist of conventions rather than a measure of real data: a patronymic read as a family name (`Hassan bin Yusof`), a title read as a given name (`U Kyaw Win`). It uses the Wikidata source's `patronymic`, written with its connective (`bin Yusof`), and adds `honorific`, a word of address written after the name (`Bey`, `ssi`, `san`), scored inside `suffix` as nameparser reads `さん` (Derek, 2026-10-16). A row no field can state honestly is left out of the fields file and stays in the corpus, which the snapshot still watches: the kunya `Abu Karim`, the `-né` married form `Szabóné Kiss Éva`, and `arap`/`ole`/`wa`, whose last word is a parent's name or a family name depending on the family. Such a row says why in its `unstated`, and the builder refuses a row with neither `fields` nor `unstated`, so this fields file states a subset of its corpus without a row's fields being lost unseen. Being hand-written, it is also held to two checks the generated sources pass anyway: its fields partition each text word for word (`test_the_fields_partition_their_rendering`), and the scorer raises on a field name no rule reads, for every source.