OPENNLP-1879: Introduce Gazetteer and geocoder API with a bundled Natural Earth implementation - #1154
Draft
krickert wants to merge 21 commits into
Draft
OPENNLP-1879: Introduce Gazetteer and geocoder API with a bundled Natural Earth implementation#1154krickert wants to merge 21 commits into
krickert wants to merge 21 commits into
Conversation
mawiesne
marked this pull request as draft
July 8, 2026 04:32
krickert
force-pushed
the
gazetteer-api
branch
4 times, most recently
from
July 8, 2026 19:05
6e5a2d8 to
6e8ece7
Compare
Contributor
|
Hi @krickert - as mentioned on Slack I currently dont have the time for a manual review but I just let Fable do a comprehensive review on this PR. Here is the result: Correctness / API design
Minor
Human review will follow. |
Contributor
Author
|
Thanks, the data finding especially. Addressed in 224cfe6:
|
Contributor
Author
|
I piggybacked a simple CDI bean and you can implement a dependent loader. Was easy, kept the format consistent. |
Contributor
? |
Contributor
Author
Got CDI vs SPI mixed up. I made the Gazetter data load in a generic SPI bean so people can bring their own. But that was over-scoping + a more generic one can be used all over the app. I removed it to abstract it more across the app. I'll introduce the idea in JIRA / dev email after a few more tickets land. For now it's a generic interface. |
mawiesne
changed the title
Define the location-intelligence seam in opennlp-api: GeoPoint (WGS84, range-validated), GazetteerEntry (immutable, defensive copies, provenance-tagged attribute map), AttributeValue, Gazetteer (name, id, and ISO 3166-1 alpha-2 region lookup), GeoResolution (mention span in original document coordinates, confidence in [0, 1]), and Geocoder. The shape is deliberately generic: neutral field names, source-scoped identifiers, and dataset-specific extras carried as provenance-tagged attribute values, so bundled, downloaded, and user-supplied datasets all sit behind the same interface. Attribute keys for stable external identifiers (fips, geoid, zcta, wikidata, geonames, whosonfirst) follow a documented convention so downstream enrichment data can join through an entry without any dataset being bundled. The country code is the join key shared with the emoji annotation layer. No new dependencies in opennlp-api.
New opennlp-extensions/opennlp-geo module (spellcheck precedent: the core jars stay lean) holding the reference Gazetteer implementation. BundledGazetteer reads a semicolon separated populated-places table (source;recordId;name;altNames;lat;lon;iso2;containment;population; featureClass;attributes), cursor-parsed line by line with fail-loud IllegalArgumentException naming the resource and line for malformed rows, loaded lazily once and immutable after load. Both indexed names and queries fold through the same matching chain (UAX apache#29 word segmentation, then NFC, case fold, accent fold per token), so case, accent, and hyphenation variants of a name produce the same match key. Lookup results rank by population descending, then a CITY over ADMIN over POI feature-class prior, then source and record id, giving a deterministic total order. byRegion serves the ISO 3166-1 alpha-2 join key with the most populous entry for the region. The bundled data table itself follows in the next commit; tests here run against crafted in-memory rows.
Adds the project-authored gazetteer table derived from the Natural Earth 1:10m Populated Places dataset (public domain): 7342 rows, one per populated place, with the NE id as the stable record id, the transliterated ASCII name (NAMEASCII), alternate names, WGS84 point coordinates, ISO 3166-1 alpha-2 country code, the admin-1 name as the v1 containment element, POP_MAX, featureClass CITY, and the external identifiers Natural Earth publishes (wikidata, geonames, whosonfirst) as provenance-tagged attributes. The full derivation record (source, mirror commit, extraction date, field mapping, public-domain dedication) is in the file header. The table is deliberately pure ASCII in v1: non-ASCII primary names are carried as the ASCII transliteration Natural Earth provides, with the Unicode forms omitted; the folded matching chain keeps accented queries working against the ASCII rows. The audit test enforces the header's claims row by row: row-count floor, pure-ASCII bytes, no duplicate ids, country-code shape (the emoji-layer join key), lat/lon range, the attribute-key whitelist, and deterministic lookups. LICENSE gains the Natural Earth public-domain section; per the ASF guidance for public-domain data there is no NOTICE entry.
Resolves each location mention independently: candidates come from the gazetteer lookup and are re-ranked here under the module's shared deterministic order (population descending, CITY over ADMIN over POI on population ties, then source and record id), so the result never depends on the gazetteer's return order. Unresolved mentions are omitted; results stay aligned to the input spans in input order. Confidence is an explicitly documented heuristic prior, not a probability: 0.9 for a single candidate, otherwise 0.5 + 0.4 * (p1 - p2) / (p1 + p2), monotonic in the relative population separation between winner and runner-up, 0.5 on a dead tie, and never 1.0. Document-context-aware scoring (co-occurring mentions constraining each other) is the named follow-up as a second Geocoder implementation; resolve() already receives the whole document for that reason. The candidate order moves into a shared package-private CandidateRanking used by both BundledGazetteer and the geocoder.
… gazetteer seam The bundled Natural Earth table is now produced by a committed generation script (opennlp-geo/dev/derive-populated-places.py) that detects and handles the encoding damage the upstream distribution itself carries. The canonical name derives from the upstream display name folded to ASCII, with the upstream transliteration as a fallback and, where it differs, as an alternate name; values that announce damage with '?' are omitted and counted, values verified against accent-bearing upstream siblings or documented place names are repaired through audited tables in the script, and one-character corruptions of a name at an accented position are dropped. The script hard-fails on anomalies it does not recognize, so a data refresh forces a reviewed decision. The regenerated table changes 181 of 7342 rows. The data file now carries the derivation record and a LICENSE pointer instead of an ASF source header, is listed in rat-excludes, and the Natural Earth public-domain section is added to the binary distribution LICENSE alongside the existing root LICENSE section. On the seam, Gazetteer.lookup, byId, byRegion and Geocoder.resolve declare IOException so database-backed and remote implementations have a portable failure channel, matching the EntityLinker precedent, and both interfaces now cross-reference opennlp.tools.entitylinker with guidance on which contract new consumers should target. The feature-class and attribute-key vocabularies are published as constants on GazetteerEntry, the UNSPECIFIED provenance marker as a constant on AttributeValue, and CandidateRanking matches on the constants so any gazetteer using them gets the feature-class prior. The population zero sentinel is documented as a deliberate choice inherited from upstream conventions, and javadoc references to an unshipped annotation layer are rephrased to the underlying ISO 3166-1 join-key fact. BundledGazetteer gains a public fromEntries factory so callers can index their own rows without the singleton, and a name that folds to an empty match key now fails loud at load time instead of loading an unreachable record. PopulationPriorGeocoder computes confidence in double with clamping so adversarial populations near Long.MAX_VALUE cannot break the GeoResolution contract, and skips the per-mention copy and sort for single-candidate lookups. The audit test enforces the header row count exactly, rejects mojibake artifact patterns and damage markers across every name, alternate name, and containment value, and asserts every name yields a non-empty fold key; parser fail-loud coverage now includes duplicate attribute keys, empty pipe-list elements, and out-of-range longitude, with use-case tests for the custom-entries factory and the third-party feature-class prior.
…ionale commentary Applies the review conventions from the OPENNLP-1869 review: the bundled gazetteer uses the holder idiom instead of volatile double-checked locking, private helpers gain javadoc, and rationale text shrinks to contract sentences.
…anking constructor
…geocoder Adds GeoNamesGazetteer, a Gazetteer over a user-supplied file in the GeoNames main table format, indexing canonical, ASCII, and alternate names case-insensitively with population-ranked candidates; the ASCII column is what makes accent-free queries hit accented places. The file is downloaded by the caller and nothing is bundled, per the LEGAL-732 guidance, so coverage scales from the bundled table to the full city extracts without a licensing question. Adds SpatialCoherenceGeocoder: a two-pass resolver that starts from every mention's population-prior pick and then moves each ambiguous mention to the candidate with the smallest mean great-circle distance to the document's other mentions, so a Paris next to Dallas and Houston lands in Texas while a lone Paris or a Paris next to London stays in France. Confidence reflects the separation between the best and second-best choice. Test fixtures are project-authored synthetic rows.
…gazetteer index Adds OvertureGazetteer, a Gazetteer over a division table derived from Overture Maps data with the new dev/derive-overture-divisions.py script: countries, regions, counties, local administrative areas, and localities with a population floor, so mentions like Australia or Bavaria resolve, which place-only gazetteers cannot do. The upstream data is CDLA-Permissive-2.0, classified Category A in LEGAL-732, and is published as partitioned Parquet, which this module deliberately does not parse; the script flattens divisions into a plain tab-separated table whose header carries the derivation record, and nothing is committed until a derivation is reviewed. The name, id, and country indexing shared with the GeoNames loader moves into the package-private GazetteerIndex, so both file loaders carry only their format parsing. Test fixtures are project-authored synthetic rows.
…x the Overture divisions license to ODbL
… constants, fix test charsets, cite standards
… definition A blank check under the toolkit's whitespace definition, which unlike String.isBlank covers the no-break spaces, so annotators validating labels and identifiers share one predicate instead of each carrying a private copy. Reads whole code points; tests pin the no-break and figure spaces, the empty string, and a supplementary-plane letter.
…ent reader PlaceHierarchy walks a place's containment chain as PlaceAncestor steps; ContainmentSpine implements it over the Who's on First ancestors table and names CSV, reading both as streams with a replacing UTF-8 decoder so a malformed byte degrades one field instead of failing the load.
… the file loaders
…sions, and bounding boxes Places a public dataset does not know can now take part in geocoding. UserGazetteer loads a user-authored tab-separated file of places over the shared in-memory index, and OverlayGazetteer composes any base gazetteer with additions from a second gazetteer plus Suppression rules hiding base entries. Additions rank first, suppressions never hide additions, so a rule plus an addition with the same name replaces a base entry, and the merged view goes through the unchanged Gazetteer seam, so every consumer picks the changes up without knowing they exist. The additions side is any Gazetteer implementation, which keeps the door open for database-backed or remote sources injected by a container. A suppression rule is a name with optional country and feature-class filters, matched against the canonical and alternate names, so one rule can remove exactly the one ambiguous reading that is noise in a corpus. Rules load from a small tab-separated file or construct directly. GazetteerEntry gains an optional GeoBoundingBox, a new opennlp-api record in the RFC 7946 axis order whose contains and center methods honor the antimeridian convention of that specification. A user row may carry a bounding box instead of coordinates, in which case its point location is the box's center. Free-text metadata such as postal addresses lives in the attribute columns, carried verbatim and never parsed. The manual gains a gazetteer chapter whose overlay example is asserted verbatim by a new test, and 77 tests cover the new surface.
CrmGazetteer in the test sources is the complete, copyable implementation the manual's new bring-your-own section shows: two maps over an application-internal store joining the seam with no registration, composed over the bundled base through the overlay. CustomGazetteerExampleTest asserts the documented behavior plus the contract corners, id scoping, region-code validation, and the null guards, that every implementation must honor. The registry-free shape is also what makes an implementation a plain bean in any dependency-injection container, stated in the manual without adding any framework type to OpenNLP.
Add a geocoder section to the geo manual and GeocoderUsageExampleTest asserting the resolve workflow the chapter prints.
…bling example tests
…ment with the review conventions
…ments in messages, ship opennlp-geo - Drop StringUtil.isBlank and its test. opennlp-api already ships isUnicodeBlank, which is null safe and uses the same whitespace definition, so the added helper was a duplicate of an existing one. PlaceAncestor, Suppression, UserGazetteer, GeoNamesGazetteer and OvertureGazetteer now call isUnicodeBlank directly and lose their separate null check. - Replace the private stripped() helper in ContainmentSpine with the existing StringUtil.trimUnicodeWhitespace, which trims by that same definition, so the CSV and Who's On First readers no longer carry their own copy. - Spell argument names in IllegalArgumentException messages exactly as the signature spells them, lower case, across the geo records, the gazetteers, the geocoders and GazetteerIndex, and update the assertions in the matching tests. - Extract the CSV reader's empty pushback marker in ContainmentSpine as the NO_PENDING constant instead of a local named none, and document why it is not the -1 a Reader returns at end of input. - Narrow CandidateRanking.featureClassRank to private; nothing outside the class calls it. - Compare Suppression names with equalsIgnoreCase instead of lower casing both sides on every call, which also drops the now unused Locale import. - Fill in the Javadoc tags that were missing on BundledGazetteer.parseTable, on GazetteerIndex add, lookup and byId, on the ContainmentSpine file readers and its CsvRows.row callback, and describe the previously undocumented Node record. - Drop the "seam" metaphor from the Gazetteer and Geocoder Javadoc, the OverlayGazetteer Javadoc, the manual and the test comments, naming the interface or the method that is actually meant. - Stop promising thread safety on behalf of implementors. Gazetteer and Geocoder now state that thread safety is implementation specific, OverlayGazetteer promises it only when the composed gazetteers provide it, and the manual asks an implementor to document its own guarantee while noting that the implementations shipped here are immutable. - Correct the geo manual, which counted three file-backed implementations while the module ships four, by listing UserGazetteer with the others, and align the CrmGazetteer byId example with the compiled CrmGazetteer, which rejects source and recordId separately. - Add opennlp-geo to the root dependencyManagement block and to opennlp-distr so the module reaches the binary distribution like the other extensions.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the location-intelligence seam and its first implementation.
Contracts in opennlp-api (
opennlp.tools.geo):GeoPoint,GeoBoundingBox,GazetteerEntry,AttributeValue,Gazetteer,GeoResolution,Geocoder. Generic in shape: the dataset is an implementation, never the contract. An entry may carry an optional bounding box (RFC 7946 axis order, antimeridian-aware) next to its point location. Attributes are provenance-tagged, with a documented external-id key convention (fips,geoid,zcta,wikidata,geonames,whosonfirst) so downstream enrichment joins on stable keys, and ISO 3166-1 alpha-2 is the region join key shared with the emoji annotation layer. No new dependencies.Implementation in a new
opennlp-extensions/opennlp-geomodule:BundledGazetteerover a project-authored 7,342-row public-domain table derived from Natural Earth Populated Places: fail-loud cursor parser, lazy single load, matching through the shared NFC/case/accent folding chain.PopulationPriorGeocoder: deterministic population-plus-feature-class ranking, documented heuristic confidence, unresolved mentions omitted.UserGazetteerplusOverlayGazetteerwithSuppressionrules: user-authored places (a tab-separated file, or anyGazetteerimplementation) composed over a base, with per-name removals, so custom additions and subtractions ride the unchanged seam. Documented in a new manual chapter whose examples are asserted by tests, including a complete bring-your-ownGazetteerreference implementation kept in the test sources.LICENSE gains the Natural Earth public-domain section; audit tests enforce the data file's documented claims row by row.
Data derivation: Natural Earth 1:10m Populated Places via the nvkelso GitHub mirror (upstream commit 789c9904, VERSION 5.2.0-pre), NAMEASCII canonical, wikidata/geonames/whosonfirst ids carried as published, no fabricated fips/geoid/zcta.
Verification: api 350/0 (64 of them in the new
opennlp.tools.geopackage), opennlp-geo 170/0, 24-module reactor verify green.Follow-ups (deliberately out of this PR): opennlp-distr wiring plus the binary LICENSE section, a context-minimization
Geocoder, and the gRPC mirror after one review round.https://issues.apache.org/jira/browse/OPENNLP-1879