Skip to content

feat(titles): canonical incident names per adapter#50

Merged
nullhack merged 1 commit into
mainfrom
fix/canonical-names
Jul 22, 2026
Merged

feat(titles): canonical incident names per adapter#50
nullhack merged 1 commit into
mainfrom
fix/canonical-names

Conversation

@nullhack

Copy link
Copy Markdown
Owner

Problem

Dashboard incident titles were verbose and inconsistent — GDACS tropical cyclone titles were 100+ char run-on sentences (e.g. "Red notification for tropical cyclone BAVI-26. Population affected by Category 1 (120 km/h) wind speeds or higher is 36.778 million ."), USGS titles were "M 7.5 - 20 km ESE of Yumare, Venezuela", WHO titles varied wildly.

The dashboard's synth override (generate_dashboard_data.py:504) only fired when max_mag is not None, which failed for GDACS TC/Flood records with non-numeric severity prose — so the verbose raw name leaked straight through to canonical_name.

Fix

Standardised format produced at ingest by each adapter:

{Type} {Identifier} {Smallest-place}, {Country} {YYYY-MM-DD}

Diseases omit the Type word (disease name leads):

{Disease} {Place} {YYYY-MM-DD}

Samples (before → after)

incident_id Before After
20260624-VE-EQ Earthquake 20 km ESE of Yumare, Venezuela 2026-06 Earthquake M7.5 Yumare, Venezuela 2026-06-24
20260701-CN-TC Red notification for tropical cyclone BAVI-26. Population affected... (118 chars) Tropical Cyclone BAVI-26 China 2026-07-01
20260606-CN-FL Orange flood alert in China Flood Hunan, China 2026-06-06
20251014-ID-VO Eruption Kanlaon Volcano Kanlaon, Philippines 2025-10-14
20250905-CD-DI Ebola virus disease – Democratic Republic of the Congo Ebola Kinshasa, DR Congo 2025-09-05
20250528-XX-DI COVID-19 - Global Situation COVID-19 Global 2025-05-28
20251221-XX-DR Drought Green Drought 2025-12-21

Architecture (adapter-centric, not a one-fits-all parser)

Each adapter owns _extract_canonical_name(raw_fields, places, report_date, incident_type) -> str, called at ingest so SourceReport.name is born canonical. Shared utilities in disaster_report/_title_format.py:

  • format_title(*parts) — joins non-empty parts with single spaces
  • smallest_place(places) -> (smallest, country) — locality > subdivision > country
  • format_place(smallest, country)"smallest, country" or just "country"
  • normalise_subdivision(name) — strips accents + admin suffixes ( Sheng, SAR, Province, etc.)
  • normalise_country(name) — short aliases (DR Congo, UK, etc.)

Per-adapter extraction

  • USGS: M{mag:.1f} from raw_fields.mag + smallest_place.
  • GDACS: dispatches by incident_type — EQ magnitude from numeric severity or regex on severitytext; TC storm name from eventname or title regex; VO volcano name from title (volcano name IS the locality); FL/WF/DR omit the identifier.
  • WHO: disease short name via prefix lookup table (25 entries covering all 28 production incident_type values); place resolved from the title suffix (split on dash variants, scan_countries on the suffix) to avoid body-scan medevac noise — falls back to smallest_place(places) or "Global" when no places.

Dashboard simplification

Dropped the synth override at generate_dashboard_data.py:503-505. canonical_name now = inc["name"] directly.

Backfill migration

scripts/backfill_canonical_names.py: one-shot, per-adapter dispatch, targeted name: line replacement only (rest of YAML preserved byte-for-byte), --dry-run flag, idempotent. Data branch backfilled separately (164 reports: 67 USGS + 52 WHO + 45 GDACS).

Tests

52 new tests: 27 _title_format + 5 USGS + 10 GDACS + 10 WHO (4 new for title-suffix parsing).

Gates

  • ruff check: clean
  • pyright: 0 errors
  • mypy.stubtest: clean (10 modules)
  • pytest: full suite green

Related

  • Data branch backfill: 0ded726 chore(data): backfill canonical report names (164 reports)
  • Dashboard publish: follow-up (regenerate from backfilled data)

Replace verbose/inconsistent genesis report names (e.g. 100-char GDACS TC
sentences, 'Flood in Pakistan', 'M 7.5 - 20 km ESE of Yumare, Venezuela')
with a standardised format produced at ingest by each adapter:

    {Type} {Identifier} {Smallest-place}, {Country} {YYYY-MM-DD}

Diseases omit the Type word — the disease name leads instead:

    {Disease} {Place} {YYYY-MM-DD}

Architecture (adapter-centric, not a one-fits-all parser):
- Each adapter owns _extract_canonical_name(raw_fields, places, report_date,
  incident_type) -> str, called at ingest so SourceReport.name is born
  canonical.
- Shared utilities in disaster_report/_title_format.py: format_title,
  smallest_place (locality > subdivision > country), format_place,
  normalise_subdivision (strips accents + admin suffixes like ' Sheng'/
  ' SAR'), normalise_country (short aliases like 'DR Congo').
- _search_keys.place_token (renamed from _place_token, now public) reused
  for locality offset-stripping.

Per-adapter extraction:
- USGS: M{mag:.1f} from raw_fields.mag + smallest_place.
- GDACS: dispatches by incident_type — EQ magnitude from numeric severity
  or regex on severitytext; TC storm name from eventname or title regex;
  VO volcano name from title (volcano name IS the locality); FL/WF/DR
  omit the identifier.
- WHO: disease short name via prefix lookup table (25 entries covering
  all 28 production incident_type values); place resolved from the title
  suffix (split on dash variants, scan_countries on the suffix) to avoid
  body-scan medevac noise — falls back to smallest_place(places) or
  'Global' when no places.

Dashboard simplification: dropped the synth override at
generate_dashboard_data.py:503-505 (was f'{inc_type} {place} {YYYY-MM}'
gated on max_mag is not None — failed for GDACS TC/Flood with non-numeric
severity). canonical_name now = inc['name'] directly.

Backfill migration script scripts/backfill_canonical_names.py: one-shot,
per-adapter dispatch, targeted name: line replacement only (rest of YAML
preserved byte-for-byte), --dry-run flag, idempotent (skips WHO records
whose name already ends with report_date). Data branch backfilled
separately (164 reports: 67 USGS + 52 WHO + 45 GDACS).

Tests: 27 _title_format + 5 USGS + 10 GDACS + 10 WHO (4 new for
title-suffix parsing) = 52 new tests. Gates: ruff clean, pyright 0
errors, mypy.stubtest clean, full suite green.
nullhack added a commit that referenced this pull request Jul 22, 2026
Regenerated from data branch at 0ded726 (post-backfill of 164 canonical
report names). All daily digests now use the standardised format:
{Type} {Identifier} {Place}, {Country} {YYYY-MM-DD}.

See PR #50 for the source changes.
@nullhack
nullhack merged commit b225e32 into main Jul 22, 2026
1 check passed
@nullhack
nullhack deleted the fix/canonical-names branch July 22, 2026 08:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant