OPENNLP-1927: Expand Hunspell dictionary support

[krickert]

Extends Hunspell dictionary loading, stemming, and morphological analysis. Unsupported directives fail by default; partial loading is an explicit choice that reports the skipped directives.

The engine now covers the directives the ticket names and the further behavior a sweep of the reference implementation's own test fixtures turned up: zero-material affixes, compound restrictions, forbidden-word ordering, hidden capitalized forms, Turkic and mixed-case input, the Hungarian hyphen rule, and the reference field order of analyses. The manual lists the remaining deviations with a workaround for each.

The fixture tests keep the stems and recognition outcomes captured from Hunspell e184e22c on fixtures written for the tests; analyses are compared in HunspellCompletionTest and assert OpenNLP results against them; no native code or process is part of the build. HunspellCompatibilityEval extends AbstractEvalTest, loads the LibreOffice en_US, de_DE_frami, and hu_HU dictionaries from hunspell/ under OPENNLP_DATA_DIR with MD5 checks, and compares the stems and recognition of 49 inputs with the captured Hunspell results. The dictionaries are in opennlp-data.zip on nightlies as of 2026-09-07. HunspellRealDictionaryTest, a property-gated test in opennlp-runtime, is removed; its assertions moved into that eval.

No dictionaries or native libraries are bundled. The shared Stemmer interface is unchanged.

HunspellStemmer.analyze is package-private; the public API returns stems. Loading defaults to STRICT, a change since M6 noted in the JIRA release note.

Verification: opennlp-runtime 3,134 tests with checkstyle; the eval, 13 tests, against the local data directory and the published archive.

OPENNLP-1927

[rzo1]

Two structural changes needed before this can go in.

1. No C/C++ sources in the repo

Drop dev/hunspell-reference.cc. Maven never compiles it, CI never runs it, and no toolchain in this project covers it, so it will go stale without anyone noticing. It also exists only to link against an external GPL/LGPL/MPL project.

The method is worth documenting. Keep dev/README-hunspell-dictionaries.md and describe how you produced the reference output: the pinned Hunspell revision e184e22c..., the C API functions you called, and the g++ command. Anyone who wants to reproduce it can write those 30 lines themselves.

That means HunspellTestSupport in opennlp-runtime also has to lose the ProcessBuilder call and the opennlp.hunspell.reference property. Unit tests in opennlp-core must not fork an external native binary. The fixture dictionaries and the expected outputs you derived from the reference are the right thing to commit.

2. Eval tests should follow the existing pattern

HunspellCompatibilityEval defines its own contract: three custom system properties (opennlp.hunspell.dict.dir, opennlp.hunspell.eval.words.dir, opennlp.hunspell.reference), Assumptions-based skipping, and required input files that live nowhere we control.

The other evals in that module extend AbstractEvalTest and resolve inputs through getOpennlpDataDir() (-DOPENNLP_DATA_DIR=... -Peval-tests), checking data with verifyFileChecksum / verifyDirectoryChecksum. Do the same here:

• class HunspellCompatibilityEval extends AbstractEvalTest
• Read dictionaries from new File(getOpennlpDataDir(), "hunspell/...") instead of a custom property.
• Keep the SHA-256 pins, expressed the way the module already does it.
• Remove the Assumptions skips. Evals run under -Peval-tests against data that is expected to be there, and fail if it isn't.

The LibreOffice dictionaries and word lists go into opennlp-data.zip on nightlies.apache.org/opennlp/, under a hunspell/ subdirectory, not into a directory each contributor assembles by hand. Any committer can update that archive:

curl -u your_asf_username -T ./opennlp-data.zip "https://nightlies.apache.org/opennlp/"

Note that the checksum file published next to the zip has to be regenerated and uploaded in the same step, otherwise the eval job fails against the new archive. See https://nightlies.apache.org/authoring.html. If you don't have an ASF account, list the exact files and revisions and one of us will upload them. Nothing is downloaded at build time either way.

Revert the addition to opennlp-docs/src/docbkx/evaltest.xml. Once the eval uses OPENNLP_DATA_DIR, that chapter already covers it. Hunspell specifics belong in stemmer.xml.

One more thing: HunspellRealDictionaryTest is removed (126 lines) without a note in the description. What covers that now?

[krickert]

There was a lot I wasn't sure about which is why I kept it in draft, thanks for chiming in.

The driver source and the test support that forked it are removed. The README describes how the reference results were produced: Hunspell e184e22c, the C API calls, the input and output convention, and the build line. The fixture tests keep the captured reference outcomes and assert OpenNLP results against them; each intended deviation is named with its reason from the manual, and a named fixture must still differ, so the data cannot go stale silently.

HunspellCompatibilityEval now extends AbstractEvalTest, loads from hunspell/ under OPENNLP_DATA_DIR, verifies MD5 digests with verifyFileChecksum, and has no Assumptions or custom
properties. It compares 49 inputs with captured reference results; the word-list variant is dropped. The evaltest.xml addition is reverted.

For opennlp-data.zip, under hunspell/, from LibreOffice dictionaries commit 32b006a2c22a4ac7e8ed3f03346f7b3d85a970a4: en_US.aff, en_US.dic, README_en_US.txt, de_DE_frami.aff, de_DE_frami.dic, README_de_DE_frami.txt, hu_HU.aff, hu_HU.dic, README_hu_HU.txt. The MD5 pins in the eval match these files.

HunspellRealDictionaryTest was a property-gated test in opennlp-runtime against the same three dictionaries. Its stem and compound assertions moved verbatim into HunspellCompatibilityEval.expectedInflections, which is where external data belongs; I will add that to the description.

I feel like I have to do one more pass then flip to ready.

[krickert]

@rzo1 I'll give the upload a shot :) I have the zip, a SHA, and a decent connection speed - should work for me. I want to do this because I'll probably have to start adding data to this more than once. If it doesn't work I'll let ya know.

[mawiesne]

We will target this PR / topic with the 3.0.0 release, not M6.

[krickert]

@mawiesne @rzo1 I did add the hunspell data to the eval tests too. It works locally + worked on the other eval build - but I'll kick off an eval build for this today too

[krickert]

Turning to draft, I'm adding tests and doing a few minor changes.

[rzo1]

Thanks, the points from my last review are addressed: no native sources, no forked binary in unit tests, the eval follows the AbstractEvalTest pattern, and the data is in opennlp-data.zip.

Still needed before merge:

1. Fixture provenance. HunspellCompatibilityTest says the fixtures are independently written and not excerpts. Several are taken from Hunspell's tests/ at e184e22c, e.g. compoundforbid.aff/.dic and gh106.aff/.dic are identical, and checkcompoundpattern2, onlyincompound2, ph2, opentaal_forbiddenword1 and allcaps are close. HunspellDictionary even links tests/compoundforbid.aff. Hunspell is MPL/GPL/LGPL. These snippets are small, but I don't want to guess. Please write our own fixtures (own words and flags, same rule being tested), or name the source of each one so we can decide. In both cases the Javadoc claim has to go.
2. Behavior change since M6. In 3.0.0-M6 load(...) ignored unsupported directives. Now it defaults to STRICT and throws IOException, so dictionaries that loaded in M6 can fail now. That's fine for 3.0.0, but it needs a line in the JIRA as release note.
3. HunspellStemmer#analyze. Public, returns raw Hunspell field strings, and only tests and the eval use it. This is frozen with 3.0.0. Either make it package-private for now, or explain in the JIRA why this is the API we want.
4. Allocations in the stemming path. See the inline comments.

Also: HunspellDictionary#parseAffix is 175 lines. Please split it per directive group.

Please also post the eval build link for the current head.

[krickert]

rzo1 - this rung's for you. All points are addressed on 8539b45 (merged with main today); eval run: https://ci-builds.apache.org/job/OpenNLP/job/eval-tests-configurable/84/.

AI summary:

1. Fixture provenance. Every fixture that duplicated a file under Hunspell's tests/ (compoundforbid, gh106, checkcompoundpattern2, onlyincompound2, ph2, opentaal_forbiddenword1, allcaps, and the near ones allcaps2, hu, checksharpsutf, base_utf, morph, limit-multiple-compounding) now uses its own words and flags for the same rule, in both HunspellCompatibilityTest and HunspellCompletionTest. The Javadoc claim is removed; the class comment says the fixtures are written for the test and the outcomes are recorded from Hunspell. The link to tests/compoundforbid.aff and the other pinned links are out of the Javadoc; the revision is in the README only. All 131 fixture outcomes were re-recorded against e184e22c with the driver the README describes.
2. M6 behaviour change. Release note added to OPENNLP-1927.
3. analyze. Package-private for 3.0.0. I filed Trim whitespace from input string. #1 for a typed analysis result, which is where it becomes public again. The eval compares stems and recognition; analyses are compared on the in-package fixtures in HunspellCompletionTest. The manual no longer documents analyze.
4. Allocations. int[] overloads for validStandalone, mayStand, supportsPart, supports, supportsCrossProduct, forbidsAffixed and noteForbidden, so the per-candidate List.of is gone at all twelve sites. lookup returns the stored lists and only builds one when a word has both listed and hidden entries. One correction: the lists were ArrayLists, not immutable; they are now copied with List.copyOf at load. variants(input) is computed once. BREAK anchors are parsed at load into a WordBreak record. The literals are constants.
5. Wording. "the reference implementation" is out of the Javadoc and the manual; each site describes the behaviour.
6. README. The undone paragraph was wrong. Both implementations stem it the same way and the eval had it as an exact match. Replaced.
7. parseAffix is 45 lines; the groups are readFlagDeclaration, readCompoundSetting, readGeneralSetting.
8. load Javadoc aligned across the four overloads.
9. Eval. println dropped; checksums verified and dictionaries loaded once in @BeforeAll.