Repository navigation
Fix Pynini compatibility on Python 3.14 - #531
janbernloehr wants to merge 6 commits into
Conversation
Use pynini 2.1.7 on Python >= 3.14 and preserve 2.1.6.post1 on older interpreters in both runtime and test requirements. Fixes NVIDIA#530 Signed-off-by: Jan Bernloehr <jbernloehr@nvidia.com>
b8bb72c to
36e2321
Compare
Give the alphanumeric fallback a slightly higher weight in deterministic mode so equal-cost paths do not change the reading of 133-ABC when upgrading Pynini. The existing serial regression test covers this case. Signed-off-by: Jan Bernloehr <jbernloehr@nvidia.com>
Signed-off-by: Jan Bernloehr <jbernloehr@nvidia.com>
Signed-off-by: Jan Bernloehr <jbernloehr@nvidia.com>
|
Had a test set suite run for previous python compatibility. We can move up pynini for all, but need a pin for 3.8. Here's the readout: Python 3.8 now tries to build Pynini from source. Pynini 2.1.7 has no cp38 wheel, but its sdist allows With these markers, plain Test results. All 21 CPU test groups (
|
Signed-off-by: Jan Bernloehr <jbernloehr@nvidia.com>
|
Aight my overnight tests came back, this causes a regression in other languages. tldr; string_map order changed between releases and it break Viet tests along with Italian. As you're working on Italian for a separate branch we can leave it to that but we should catch the Viet break. 1. Sparrowhawk: Vietnamese and Italian TN regress with 2.1.7 exportsI exported every non-English config that has a Sparrowhawk test, from this branch with Pynini 2.1.7 and from
These are the same mechanism as the English cases you fixed: exact weight ties. Enumerating every path of the composed lattices (both exports, both Pynini versions):
Why 2.1.7 flips them: Which configurations give which reading
The Pynini column is a bare Could we add the same kind of priority weight to the |
Signed-off-by: Jan Bernloehr <jbernloehr@nvidia.com>
What does this PR do ?
Fix Linux x86_64 installation on Python 3.14 while preserving the compatible Pynini wheel on Python 3.8. Apply the same environment markers in runtime and test requirements:
The README Conda instructions also document the Python 3.8 pin.
Fixes #530.
The unconditional 2.1.6.post1 pin forces Python 3.14 installations to build Pynini from source because that release has no compatible wheel. Without OpenFst development headers, the build fails with
fst/util.h: No such file or directory. Pynini 2.1.7 provides a CPython 3.14 Linux x86_64 wheel, but has no CPython 3.8 wheel. Its source archive allows Python 3.8, so the explicit old-version pin prevents normal pip from selecting a source build on that version.The upgrade also changes which equally weighted grammar paths are selected. Add deterministic priorities to preserve the existing English readings of
133-ABCand the URL suffixprogram(which otherwise becomesproGRAMin Sparrowhawk), and Vietnamese ordinal and fraction exceptions such asthứ 1→thứ nhất,thứ 4→thứ tư, and3/4→ba phần tư. Nine new Vietnamese tests require a unique best rewrite, exposing ties that ordinary shortest-path normalization can hide. Existing Sparrowhawk fixtures cover the deployment regressions. Nondeterministic grammar behavior is unchanged.Matching markers in both requirements files keep runtime,
test, andalldependencies consistent. Quote the Conda upper bound so the shell does not interpret<as redirection.Packaging verification
On Linux x86_64 / Python 3.8.20, ordinary
pip installof the Pynini entries extracted from each requirements file installedpynini-2.1.6.post1-cp38-cp38-manylinux_2_28_x86_64.whl. Source builds were allowed: neither--only-binarynor--prefer-binarywas used. Both pip installation reports confirm the wheel URL and version, and the installed library imports and executes an FST composition successfully.Manually resolved the Pynini entries extracted from each changed requirements file against public PyPI, targeting
x86_64-manylinux_2_28with source builds disabled:The unpatched
pynini==2.1.6.post1requirement fails the same Python 3.14 check with “no usable wheels.” The built package'sRequires-Distmetadata preserves both Python-version markers for runtime dependencies and thetestandallextras.To reproduce the resolver check from the patched checkout:
Manual grammar validation
On November 19, repeated all 22 CPU test groups with Linux x86_64 / Python 3.13.12 / Pynini 2.1.7: 13,533 passed, 4 known baseline failures, no errors or skips. This verifies the newer Pynini version on Python below 3.14. The subsequent Python 3.8 marker adjustment changes no grammar source or dependency selection on Python 3.9+.
Earlier validation installed the checkout with its
testextra on Linux x86_64 / Python 3.14.3 / Pynini 2.1.7 and ran all 22 CPU test groups separately: 13,533 passed, 4 failed. The four failures are Italian electronic-address test-mode mismatches addressed separately in #533; all four also reproduce on unchanged upstreamddadfb2awith Python 3.13.12 / Pynini 2.1.6.post1 (48 passed, 4 failed).The complete English suite was repeated after both grammar corrections: 1,615 passed on each dependency version. The aggregate above combines the broad dependency-upgrade sweep with this final English rerun. Kinyarwanda passed all 100 cases without the optional cache flag; enabling caching causes the same collection error on upstream. Optional audio-based expansion and model downloads were not enabled.
English Sparrowhawk with the same grammar source: 39/39 test functions passed across cased TN and lower-cased/cased ITN, using exported Pynini 2.1.7 grammars and the repository's deployment container and shell tests. All 17 TN functions were repeated after both corrections; the unchanged ITN functions passed in the preceding run. Both affected TN groups (serial and electronic) also passed with corrected grammars exported using Pynini 2.1.6.post1. Vietnamese Sparrowhawk validation is described below. Other non-English Sparrowhawk exports, Python versions after 3.14, and other platforms were not tested here.
Vietnamese validation on November 22 used fresh caches and FAR exports with the final grammar source: 934/934 Python tests passed on each of Pynini 2.1.6.post1 and 2.1.7. All 472/472 Vietnamese TN fixture records passed through Sparrowhawk on each version, including final records without trailing newlines that the upstream shell reader skips. The baseline passes 472/472 on 2.1.6.post1 and fails exactly the ten reported ordinal/fraction/measurement records on 2.1.7. Seven of the nine new unique-best-rewrite cases fail before the fix on either version and all nine pass afterward. Nondeterministic ordinal/fraction FSTs remain byte-identical before and after on each version. The full multilingual CPU suite was not repeated for this Vietnamese-only change; the earlier four Italian failures remain separate in #533.
If #474 lands first, both markers need to move into its
pyproject.tomlruntime dependencies andtestextra.Before your PR is "Ready for review"
Pre checks:
Checked items below have been reviewed and are either satisfied or explicitly not applicable to this compatibility fix.
git commit -sto sign. — All commits have matching authors andSigned-off-bytrailers.pytestor (if your machine does not have GPU)pytest --cpufrom the root folder (given you marked your test cases accordingly@pytest.mark.run_only_on('CPU')).bash tools/text_processing_deployment/export_grammars.sh --MODE=test ...pytestand Sparrowhawk here. — No new feature. Existing English Python/Sparrowhawk fixtures and Vietnamese Sparrowhawk fixtures cover the regressions; nine new Vietnamese Python cases detect ambiguous best rewrites.__init__.pyfor every folder and subfolder, includingdatafolder which has .TSV files? — N/A: no new folders.Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.to all newly added Python files? — N/A: no Python files added.Copyright 2015 and onwards Google, Inc.. See an example here. — N/A: no copied code.try import: ... except: ...) if not already done. — No import guards added.PR Type:
If you haven't finished some of the above items you can still open "Draft" PR.
Prepared with assistance from OpenAI Codex (GPT-6).