Skip to content

Staging/hi itn v3 - #498

Closed
mgrafu wants to merge 8 commits into
mainfrom
staging/hi_itn_v3
Closed

mgrafu wants to merge 8 commits into
mainfrom
staging/hi_itn_v3

Conversation

@mgrafu

@mgrafu mgrafu commented Sep 21, 2026

Copy link
Copy Markdown
Collaborator

What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

Before your PR is "Ready for review"

Pre checks:

  • Have you signed your commits? Use git commit -s to sign.
  • Do all unittests finish successfully before sending PR?
    1. pytest or (if your machine does not have GPU) pytest --cpu from the root folder (given you marked your test cases accordingly @pytest.mark.run_only_on('CPU')).
    2. Sparrowhawk tests bash tools/text_processing_deployment/export_grammars.sh --MODE=test ...
  • If you are adding a new feature: Have you added test cases for both pytest and Sparrowhawk here.
  • Have you added __init__.py for every folder and subfolder, including data folder which has .TSV files?
  • Have you followed codeQL results and removed unused variables and imports (report is at the bottom of the PR in github review box) ?
  • Have you added the correct license header Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. to all newly added Python files?
  • If you copied nemo_text_processing/text_normalization/en/graph_utils.py your header's second line should be Copyright 2015 and onwards Google, Inc.. See an example here.
  • Remove import guards (try import: ... except: ...) if not already done.
  • If you added a new language or a new feature please update the NeMo documentation (lives in different repo).
  • Have you added your language support to tools/text_processing_deployment/pynini_export.py.

PR Type:

  • New Feature
  • Bugfix
  • Documentation
  • Test

If you haven't finished some of the above items you can still open "Draft" PR.

* Hi ITN: implement Roman semiotic class

Add an inverse-text-normalization Roman numeral class for Hindi that
converts spoken numbers following a fixed set of context key words
(अध्याय / खंड / खण्ड / कक्षा) into Roman numerals.

  अध्याय तीन  -> अध्याय III
  कक्षा बारह  -> कक्षा XII

Conversion is restricted to these predictable contexts; regnal/papal and
product names are a documented limitation. Mirrors the TN Roman class
structure (data/__init__.py, license headers, pytest + sparrowhawk tests).

Signed-off-by: Mayuri S <mayuris@nvidia.com>

* Hi ITN Roman: address review feedback

Signed-off-by: Mayuri S <mayuris@nvidia.com>

* Hi ITN Roman: drop strip in test, add boundary cases, 4000 Arabic fallback, bump cache

Signed-off-by: Mayuri S <mayuris@nvidia.com>

---------

Signed-off-by: Mayuri S <mayuris@nvidia.com>
* feat(hi): add Hindi ITN electronic class

Signed-off-by: Mayuri S <mayuris@nvidia.com>

* chore: remove laptop setup guide and temp diag scripts

Signed-off-by: Mayuri S <mayuris@nvidia.com>

* chore: remove scratch helper script

Signed-off-by: Mayuri S <mayuris@nvidia.com>

* chore(hi): remove percentage class (out of scope for electronic PR)

Signed-off-by: Mayuri S <mayuris@nvidia.com>

* chore(hi): remove leftover percentage tagger and verbalizer

Signed-off-by: Mayuri S <mayuris@nvidia.com>

* style: apply black and isort formatting

Signed-off-by: Mayuri S <mayuris@nvidia.com>

* hi ITN: TSV-driven electronic refactor + serial class + review fixes

Signed-off-by: Mayuri S <mayuris@nvidia.com>

* hi ITN: move alnum to serial, reduce weights, punctuation postprocessor

Signed-off-by: Mayuri S <mayuris@nvidia.com>

* updating fixes

Signed-off-by: Mayuri S <mayuris@nvidia.com>

* hi ITN: updated the changes

Signed-off-by: Mayuri S <mayuris@nvidia.com>

* hi ITN: move telephone changes to a separate PR

Signed-off-by: Mayuri S <mayuris@nvidia.com>

* hi ITN: use exact match in serial and electronic tests

Signed-off-by: Mayuri S <mayuris@nvidia.com>

---------

Signed-off-by: Mayuri S <mayuris@nvidia.com>
Signed-off-by: mayuris-00 <mayuris@nvidia.com>
* hi ITN: add IP address support to telephone, dedupe TSV loads

Signed-off-by: Mayuri S <mayuris@nvidia.com>

* hi ITN: add shared grammar utils, symbols TSV, and CI cache bump

Signed-off-by: Mayuri S <mayuris@nvidia.com>

* chore: retrigger DCO check

Signed-off-by: Mayuri S <mayuris@nvidia.com>

---------

Signed-off-by: Mayuri S <mayuris@nvidia.com>
Signed-off-by: Mayuri S <mayuris@nvidia.com>
* hi ITN: add math operations and telephone extensions

Signed-off-by: Mayuri S <mayuris@nvidia.com>

* Updated the test cases

Signed-off-by: Mayuri S <mayuris@nvidia.com>

* hi ITN: require single digit words for contextless short numbers

Signed-off-by: Mayuri S <mayuris@nvidia.com>

---------

Signed-off-by: Mayuri S <mayuris@nvidia.com>
Signed-off-by: Mayuri S <mayuris@nvidia.com>
Signed-off-by: Mai Anh <palasek182@gmail.com>
Comment thread tests/nemo_text_processing/hi/test_serial.py Fixed

@folivoramanh folivoramanh left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed this as an integration PR rather than re-litigating the eight feature PRs it carries. Full Hindi suite passes 1230/1230 on the merge result, which covers the unchecked unit-test box in the description.

Integration checks are clean:

  • Nothing from main is lost — git log main ^HEAD is empty, so the branch is fully current after 18d10e83. Only inverse_normalize.py changes outside hi/ (+15/-0), and the Jenkinsfile keeps main's JA_/KO_ cache bumps while bumping HI_TN_CACHE.
  • Sparrowhawk wiring is complete — 15 testITN* entries with 15 matching test_cases_*.txt, including the three new classes (electronic, roman, serial).
  • The new post_process hook is genuinely applied, not dead code — hi is already in the hardcoded ["en", "hi", "ja", "vi"] list in normalize.py, and the flag toggles as intended:
    मुझे एक सौ रुपये चाहिए ।   post_process=True  -> मुझे ₹१०० चाहिए।
                               post_process=False -> मुझे ₹१०० चाहिए ।
    

should-fix · verified — nemo_text_processing/inverse_text_normalization/hi/data/electronic/__init__.py and nemo_text_processing/inverse_text_normalization/hi/data/serial/__init__.py are both 0 bytes, while every sibling carries the licence header (hi/data/numbers/__init__.py is 610 bytes, starting # Copyright (c) 2024, NVIDIA CORPORATION.). CONTRIBUTING requires the header on every new Python file, and these are the only two added .py files missing it. (Commenting here rather than inline because an empty added file has no diff hunk to anchor to.)

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Signed-off-by: Mai Anh <palasek182@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants