Skip to content

Fix Pynini compatibility on Python 3.14 - #531

Open
janbernloehr wants to merge 6 commits into
NVIDIA:mainfrom
janbernloehr:fix/pynini-python314
Open

janbernloehr wants to merge 6 commits into
NVIDIA:mainfrom
janbernloehr:fix/pynini-python314

Conversation

@janbernloehr

@janbernloehr janbernloehr commented Nov 11, 2026 •

Copy link
Copy Markdown

What does this PR do ?

Fix Linux x86_64 installation on Python 3.14 while preserving the compatible Pynini wheel on Python 3.8. Apply the same environment markers in runtime and test requirements:

pynini==2.1.6.post1; python_version < "3.9"
pynini<=2.1.7; python_version >= "3.9"

The README Conda instructions also document the Python 3.8 pin.

Fixes #530.

The unconditional 2.1.6.post1 pin forces Python 3.14 installations to build Pynini from source because that release has no compatible wheel. Without OpenFst development headers, the build fails with fst/util.h: No such file or directory. Pynini 2.1.7 provides a CPython 3.14 Linux x86_64 wheel, but has no CPython 3.8 wheel. Its source archive allows Python 3.8, so the explicit old-version pin prevents normal pip from selecting a source build on that version.

The upgrade also changes which equally weighted grammar paths are selected. Add deterministic priorities to preserve the existing English readings of 133-ABC and the URL suffix program (which otherwise becomes proGRAM in Sparrowhawk), and Vietnamese ordinal and fraction exceptions such as thứ 1 → thứ nhất, thứ 4 → thứ tư, and 3/4 → ba phần tư. Nine new Vietnamese tests require a unique best rewrite, exposing ties that ordinary shortest-path normalization can hide. Existing Sparrowhawk fixtures cover the deployment regressions. Nondeterministic grammar behavior is unchanged.

Matching markers in both requirements files keep runtime, test, and all dependencies consistent. Quote the Conda upper bound so the shell does not interpret < as redirection.

Packaging verification

On Linux x86_64 / Python 3.8.20, ordinary pip install of the Pynini entries extracted from each requirements file installed pynini-2.1.6.post1-cp38-cp38-manylinux_2_28_x86_64.whl. Source builds were allowed: neither --only-binary nor --prefer-binary was used. Both pip installation reports confirm the wheel URL and version, and the installed library imports and executes an FST composition successfully.

Manually resolved the Pynini entries extracted from each changed requirements file against public PyPI, targeting x86_64-manylinux_2_28 with source builds disabled:

Target Python Runtime requirements Test requirements
3.8 2.1.6.post1 2.1.6.post1
3.9, 3.10, 3.11, 3.12, 3.13, 3.14 2.1.7 2.1.7

The unpatched pynini==2.1.6.post1 requirement fails the same Python 3.14 check with “no usable wheels.” The built package's Requires-Dist metadata preserves both Python-version markers for runtime dependencies and the test and all extras.

To reproduce the resolver check from the patched checkout:

for requirements in requirements/requirements.txt requirements/requirements_test.txt; do
  for version in 3.8 3.9 3.10 3.11 3.12 3.13 3.14; do
    grep '^pynini' "$requirements" | uv pip compile - \
      --python-version "$version" \
      --python-platform x86_64-manylinux_2_28 \
      --only-binary :all:
  done
done

Manual grammar validation

On November 19, repeated all 22 CPU test groups with Linux x86_64 / Python 3.13.12 / Pynini 2.1.7: 13,533 passed, 4 known baseline failures, no errors or skips. This verifies the newer Pynini version on Python below 3.14. The subsequent Python 3.8 marker adjustment changes no grammar source or dependency selection on Python 3.9+.

Earlier validation installed the checkout with its test extra on Linux x86_64 / Python 3.14.3 / Pynini 2.1.7 and ran all 22 CPU test groups separately: 13,533 passed, 4 failed. The four failures are Italian electronic-address test-mode mismatches addressed separately in #533; all four also reproduce on unchanged upstream ddadfb2a with Python 3.13.12 / Pynini 2.1.6.post1 (48 passed, 4 failed).

The complete English suite was repeated after both grammar corrections: 1,615 passed on each dependency version. The aggregate above combines the broad dependency-upgrade sweep with this final English rerun. Kinyarwanda passed all 100 cases without the optional cache flag; enabling caching causes the same collection error on upstream. Optional audio-based expansion and model downloads were not enabled.

English Sparrowhawk with the same grammar source: 39/39 test functions passed across cased TN and lower-cased/cased ITN, using exported Pynini 2.1.7 grammars and the repository's deployment container and shell tests. All 17 TN functions were repeated after both corrections; the unchanged ITN functions passed in the preceding run. Both affected TN groups (serial and electronic) also passed with corrected grammars exported using Pynini 2.1.6.post1. Vietnamese Sparrowhawk validation is described below. Other non-English Sparrowhawk exports, Python versions after 3.14, and other platforms were not tested here.

Vietnamese validation on November 22 used fresh caches and FAR exports with the final grammar source: 934/934 Python tests passed on each of Pynini 2.1.6.post1 and 2.1.7. All 472/472 Vietnamese TN fixture records passed through Sparrowhawk on each version, including final records without trailing newlines that the upstream shell reader skips. The baseline passes 472/472 on 2.1.6.post1 and fails exactly the ten reported ordinal/fraction/measurement records on 2.1.7. Seven of the nine new unique-best-rewrite cases fail before the fix on either version and all nine pass afterward. Nondeterministic ordinal/fraction FSTs remain byte-identical before and after on each version. The full multilingual CPU suite was not repeated for this Vietnamese-only change; the earlier four Italian failures remain separate in #533.

If #474 lands first, both markers need to move into its pyproject.toml runtime dependencies and test extra.

Before your PR is "Ready for review"

Pre checks:
Checked items below have been reviewed and are either satisfied or explicitly not applicable to this compatibility fix.

  • Have you signed your commits? Use git commit -s to sign. — All commits have matching authors and Signed-off-by trailers.
  • Do all unittests finish successfully before sending PR? — The CPU sweep has four failures that also reproduce on unchanged upstream; see validation above. This item remains unchecked rather than claiming an entirely green suite.
    1. pytest or (if your machine does not have GPU) pytest --cpu from the root folder (given you marked your test cases accordingly @pytest.mark.run_only_on('CPU')).
    2. Sparrowhawk tests bash tools/text_processing_deployment/export_grammars.sh --MODE=test ...
  • If you are adding a new feature: Have you added test cases for both pytest and Sparrowhawk here. — No new feature. Existing English Python/Sparrowhawk fixtures and Vietnamese Sparrowhawk fixtures cover the regressions; nine new Vietnamese Python cases detect ambiguous best rewrites.
  • Have you added __init__.py for every folder and subfolder, including data folder which has .TSV files? — N/A: no new folders.
  • Have you followed codeQL results and removed unused variables and imports (report is at the bottom of the PR in github review box) ? — Reviewed the changed Python code; no unused variables or imports introduced.
  • Have you added the correct license header Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. to all newly added Python files? — N/A: no Python files added.
  • If you copied nemo_text_processing/text_normalization/en/graph_utils.py your header's second line should be Copyright 2015 and onwards Google, Inc.. See an example here. — N/A: no copied code.
  • Remove import guards (try import: ... except: ...) if not already done. — No import guards added.
  • If you added a new language or a new feature please update the NeMo documentation (lives in different repo). — N/A: no new language or feature.
  • Have you added your language support to tools/text_processing_deployment/pynini_export.py. — N/A: no new language.

PR Type:

  • New Feature
  • Bugfix
  • Documentation
  • Test

If you haven't finished some of the above items you can still open "Draft" PR.

Prepared with assistance from OpenAI Codex (GPT-6).

Use pynini 2.1.7 on Python >= 3.14 and preserve 2.1.6.post1 on older
interpreters in both runtime and test requirements.

Fixes NVIDIA#530

Signed-off-by: Jan Bernloehr <jbernloehr@nvidia.com>
Give the alphanumeric fallback a slightly higher weight in deterministic mode so equal-cost paths do not change the reading of 133-ABC when upgrading Pynini. The existing serial regression test covers this case.

Signed-off-by: Jan Bernloehr <jbernloehr@nvidia.com>
Signed-off-by: Jan Bernloehr <jbernloehr@nvidia.com>
@janbernloehr janbernloehr changed the title Fix Pynini wheel selection on Python 3.14 Fix Pynini compatibility on Python 3.14 Nov 13, 2026
Signed-off-by: Jan Bernloehr <jbernloehr@nvidia.com>
@tbartley94

Copy link
Copy Markdown
Member

Had a test set suite run for previous python compatibility. We can move up pynini for all, but need a pin for 3.8. Here's the readout:

Python 3.8 now tries to build Pynini from source. Pynini 2.1.7 has no cp38 wheel, but its sdist allows requires_python >=3.8. With pynini<=2.1.7, plain pip install on 3.8 selects the 2.1.7 sdist and compiles it, which fails unless matching OpenFst headers are installed. Before this change, 3.8 got the 2.1.6.post1 cp38 wheel. The resolver check in the description uses --only-binary :all:, which hides this. setup.py still lists 3.8 as supported. Could we keep 3.8 on the old pin with an environment marker, in both requirements/requirements.txt and requirements/requirements_test.txt?

pynini==2.1.6.post1; python_version < "3.9"
pynini<=2.1.7; python_version >= "3.9"

With these markers, plain pip on 3.8 installs the 2.1.6.post1 wheel, and the wheels-only resolve gives 2.1.7 on 3.9 through 3.14. The alternative is to drop 3.8 from the classifiers.

Test results. All 21 CPU test groups (-m "not pleasefixme" --cpu, 13,439 tests) plus audio_based_utils, on Linux x86_64. The results match main (Python 3.10, Pynini 2.1.6.post1) on every version:

Python Pynini New failures vs main
3.9, 3.10, 3.11, 3.12 2.1.7 none
3.8 2.1.6.post1 none

Signed-off-by: Jan Bernloehr <jbernloehr@nvidia.com>
@tbartley94

tbartley94 commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Aight my overnight tests came back, this causes a regression in other languages. tldr; string_map order changed between releases and it break Viet tests along with Italian. As you're working on Italian for a separate branch we can leave it to that but we should catch the Viet break.

1. Sparrowhawk: Vietnamese and Italian TN regress with 2.1.7 exports

I exported every non-English config that has a Sparrowhawk test, from this branch with Pynini 2.1.7 and from main with 2.1.6.post1 (both on Python 3.10), and ran the repo's deployment container. 27 of the 29 configs that run on both are identical. Two regress:

Config 2.1.7 export Example
vi TN 10 assertions fail thứ 4 → thứ bốn (expected thứ tư), thứ 1 → thứ một (expected thứ nhất), 3/4 → ba phần bốn (expected ba phần tư)
it TN 1 assertion fails 12:30 → dodici e mezza (expected dodici e trenta minuti)

These are the same mechanism as the English cases you fixed: exact weight ties. Enumerating every path of the composed lattices (both exports, both Pynini versions):

  • vi/taggers/ordinal.py: union(exception_graph, cardinal.graph) is unweighted, so 4 maps to both tư and bốn at 1.1, and 1 to both nhất and một. Fraction denominators tie the same way.
  • it/verbalizers/time.py: minute @ number_verbalization + insert(" minuti") | minute @ cross("30", " mezza") gives trenta minuti and mezza both weight 0. The :15 / un quarto branch has the same shape.

Why 2.1.7 flips them: pynini.string_file compiles TSV lines differently. For 0<TAB>không, 2.1.6.post1 builds 0:ε ε:ε ε:k ε:h … (read the input, then write the output), while 2.1.7 pairs labels like cross() does: 0:k ε:h …. The relation and weights are unchanged, but every grammar built from TSV data gets a different layout. The vi tagger goes from 169,635 to 120,976 states, so tied paths resolve differently. It depends on the Pynini version only: the same string_file gives the same machine on every Python version.

Which configurations give which reading
Case Export Pynini shortestpath Sparrowhawk Sparrowhawk (lookahead off)
vi thứ 4 2.1.6.post1 tư ✓ tư ✓ —
vi thứ 4 2.1.7 tư ✓ bốn ✗ bốn ✗
it 12:30 (verbalizer) 2.1.6.post1 mezza ✗ trenta minuti ✓ —
it 12:30 (verbalizer) 2.1.7 trenta minuti ✓ mezza ✗ mezza ✗
en 133-ABC (main grammars) 2.1.6.post1 ✓ ✓ one three three ✗
en 133-ABC (main grammars) 2.1.7 one three three ✗ ✓ ✓
en …-program (main grammars) 2.1.7 ✓ proGRAM ✗ proGRAM ✗
this PR's English grammars 2.1.7 ✓ ✓ ✓

The Pynini column is a bare shortestpath on the exported file; the pytest suites build the grammars in-process and pass on both versions. A tie can flip with the export layout, the decoder, or even Pynini against Sparrowhawk on the same file. That's why pytest doesn't catch these, and why your weights are the right fix: once the intended reading is weighted, every configuration agrees.

Could we add the same kind of priority weight to the vi ordinal and fraction exceptions (nhất, tư) and to the it time verbalizer (trenta minuti over mezza, and whichever :15 reading is intended)?

Signed-off-by: Jan Bernloehr <jbernloehr@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Python 3.14: nemo-text-processing cannot be installed on Linux x86_64 because of the exact pynini==2.1.6.post1 pin (no cp314 wheel)

2 participants