feat(titles): canonical incident names per adapter#50
Merged
Conversation
Replace verbose/inconsistent genesis report names (e.g. 100-char GDACS TC
sentences, 'Flood in Pakistan', 'M 7.5 - 20 km ESE of Yumare, Venezuela')
with a standardised format produced at ingest by each adapter:
{Type} {Identifier} {Smallest-place}, {Country} {YYYY-MM-DD}
Diseases omit the Type word — the disease name leads instead:
{Disease} {Place} {YYYY-MM-DD}
Architecture (adapter-centric, not a one-fits-all parser):
- Each adapter owns _extract_canonical_name(raw_fields, places, report_date,
incident_type) -> str, called at ingest so SourceReport.name is born
canonical.
- Shared utilities in disaster_report/_title_format.py: format_title,
smallest_place (locality > subdivision > country), format_place,
normalise_subdivision (strips accents + admin suffixes like ' Sheng'/
' SAR'), normalise_country (short aliases like 'DR Congo').
- _search_keys.place_token (renamed from _place_token, now public) reused
for locality offset-stripping.
Per-adapter extraction:
- USGS: M{mag:.1f} from raw_fields.mag + smallest_place.
- GDACS: dispatches by incident_type — EQ magnitude from numeric severity
or regex on severitytext; TC storm name from eventname or title regex;
VO volcano name from title (volcano name IS the locality); FL/WF/DR
omit the identifier.
- WHO: disease short name via prefix lookup table (25 entries covering
all 28 production incident_type values); place resolved from the title
suffix (split on dash variants, scan_countries on the suffix) to avoid
body-scan medevac noise — falls back to smallest_place(places) or
'Global' when no places.
Dashboard simplification: dropped the synth override at
generate_dashboard_data.py:503-505 (was f'{inc_type} {place} {YYYY-MM}'
gated on max_mag is not None — failed for GDACS TC/Flood with non-numeric
severity). canonical_name now = inc['name'] directly.
Backfill migration script scripts/backfill_canonical_names.py: one-shot,
per-adapter dispatch, targeted name: line replacement only (rest of YAML
preserved byte-for-byte), --dry-run flag, idempotent (skips WHO records
whose name already ends with report_date). Data branch backfilled
separately (164 reports: 67 USGS + 52 WHO + 45 GDACS).
Tests: 27 _title_format + 5 USGS + 10 GDACS + 10 WHO (4 new for
title-suffix parsing) = 52 new tests. Gates: ruff clean, pyright 0
errors, mypy.stubtest clean, full suite green.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Dashboard incident titles were verbose and inconsistent — GDACS tropical cyclone titles were 100+ char run-on sentences (e.g.
"Red notification for tropical cyclone BAVI-26. Population affected by Category 1 (120 km/h) wind speeds or higher is 36.778 million ."), USGS titles were"M 7.5 - 20 km ESE of Yumare, Venezuela", WHO titles varied wildly.The dashboard's synth override (
generate_dashboard_data.py:504) only fired whenmax_mag is not None, which failed for GDACS TC/Flood records with non-numericseverityprose — so the verbose raw name leaked straight through tocanonical_name.Fix
Standardised format produced at ingest by each adapter:
Diseases omit the Type word (disease name leads):
Samples (before → after)
Earthquake 20 km ESE of Yumare, Venezuela 2026-06Earthquake M7.5 Yumare, Venezuela 2026-06-24Red notification for tropical cyclone BAVI-26. Population affected...(118 chars)Tropical Cyclone BAVI-26 China 2026-07-01Orange flood alert in ChinaFlood Hunan, China 2026-06-06Eruption KanlaonVolcano Kanlaon, Philippines 2025-10-14Ebola virus disease – Democratic Republic of the CongoEbola Kinshasa, DR Congo 2025-09-05COVID-19 - Global SituationCOVID-19 Global 2025-05-28Drought GreenDrought 2025-12-21Architecture (adapter-centric, not a one-fits-all parser)
Each adapter owns
_extract_canonical_name(raw_fields, places, report_date, incident_type) -> str, called at ingest soSourceReport.nameis born canonical. Shared utilities indisaster_report/_title_format.py:format_title(*parts)— joins non-empty parts with single spacessmallest_place(places) -> (smallest, country)— locality > subdivision > countryformat_place(smallest, country)—"smallest, country"or just"country"normalise_subdivision(name)— strips accents + admin suffixes (Sheng,SAR,Province, etc.)normalise_country(name)— short aliases (DR Congo,UK, etc.)Per-adapter extraction
M{mag:.1f}fromraw_fields.mag+smallest_place.incident_type— EQ magnitude from numericseverityor regex onseveritytext; TC storm name fromeventnameor title regex; VO volcano name from title (volcano name IS the locality); FL/WF/DR omit the identifier.incident_typevalues); place resolved from the title suffix (split on dash variants,scan_countrieson the suffix) to avoid body-scan medevac noise — falls back tosmallest_place(places)or"Global"when no places.Dashboard simplification
Dropped the synth override at
generate_dashboard_data.py:503-505.canonical_namenow =inc["name"]directly.Backfill migration
scripts/backfill_canonical_names.py: one-shot, per-adapter dispatch, targetedname:line replacement only (rest of YAML preserved byte-for-byte),--dry-runflag, idempotent. Data branch backfilled separately (164 reports: 67 USGS + 52 WHO + 45 GDACS).Tests
52 new tests: 27
_title_format+ 5 USGS + 10 GDACS + 10 WHO (4 new for title-suffix parsing).Gates
ruff check: cleanpyright: 0 errorsmypy.stubtest: clean (10 modules)pytest: full suite greenRelated
0ded726 chore(data): backfill canonical report names (164 reports)