Northern Sámi TN

.pre-commit-config.yaml

@@ -22,25 +22,32 @@ ci:
 
 repos:
   - repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v4.3.0
+    rev: v6.0.0
     hooks:
       - id: check-yaml
       - id: check-case-conflict
       - id: detect-private-key
       - id: requirements-txt-fixer
 
+  - repo: https://github.com/PyCQA/flake8
+    rev: 7.3.0
+    hooks:
+    -   id: flake8
+        args:
+        - --select=W605
+
   - repo: https://github.com/PyCQA/isort
-    rev: 4.3.21
+    rev: 6.1.0
     hooks:
       - id: isort
         name: Format imports
         args: [ --multi-line=3, --trailing-comma, --force-grid-wrap=0, --use-parentheses, --line-width=119, -rc, -ws ]
         exclude: docs/
 
-  - repo: https://github.com/psf/black
-    rev: 19.10b0
+  - repo: https://github.com/psf/black-pre-commit-mirror
+    rev: 25.9.0
     hooks:
       - id: black
         name: Format code
         args: [--skip-string-normalization, --line-length=119]
-        additional_dependencies: ['click==8.0.2']
+        additional_dependencies: ['click>=8.0.2']

CHANGELOG.md

@@ -0,0 +1,51 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [r1.1.0] - 2024-08-20
+
+### Added
+
+- DE TN Electronic recognizes social media handles `@Nvidia` (#177) 
+- Japanese ITN Cardinal, Date, Decimal, Fraction, Ordinal, Time and Whitelist coverage (#141)
+
+### Fixed
+
+- Fixed Fr TN Electronic processing issue from (#166) (#181)
+- Fixed En TN Electronic processing issue from (#166) (#185) (#206) (#207)
+- Fixed It TN Electronic processing issue from (#166) (#183)
+- Fixed Hu TN Electronic processing issue from (#166) (#184)
+
+
+### Changed
+
+- De TN Time coverage allows full stop delineation (e.g. `2.10h`) (#177)
+- Es-En TN weights and data updated for unified (PnC) asr models (#143)
+- En TN Expands coverage for technical terms for TTS processing (#167)
+- En TN Money supports 'per unit' demarcation (e.g. `$20 per anum`) (#213)
+
+
+## [r1.0.2] - 2024-05-03
+
+### Added
+
+- Sentence level ZH (Mandarin Chinese) TN (#112)
+- Enabled post-processing support for Sparrowhawk TN test (#147) 
+
+### Fixed
+
+- `normalize_with_audio` text-field variable changed (#153)
+- `run_evaluate` script for ITN updated for additional languages and casing (#164)
+
+### Changed
+
+- Docstring update (#157)
+
+
+### Removed
+
+- Removed unused function from AR (Arabic) TN decimals (#165)
+

CONTRIBUTING.md

@@ -1,6 +1,6 @@
 # Contributions are welcome!
 
-We do all of NeMo-text-processing's development in the open. Contributions from the open-source community are welcome.
+We do all of NeMo-Text-Processing's development in the open. Contributions from the open-source community are welcome.
 
 
 # Pull Requests (PR) Guidelines
@@ -22,4 +22,35 @@ We do all of NeMo-text-processing's development in the open. Contributions from
 11) Optional: if you added a new language or a new feature please update the [NeMo documentation](https://github.com/NVIDIA/NeMo/blob/main/docs/source/nlp/text_normalization/wfst/wfst_text_normalization.rst) (lives in different repo).
 12) Send your PR and request a review
 
+# Notes for Language Contribution
+1) `en/graph_utils.py` and `en/utils.py` are the de facto parents for all other `graph_util` and `utils` functions, respectively. Please refrain from duplicating code logics into new `graph_utils.py` and `utils.py` files and default to imports from `en/graph_utils.py` and `en/utils.py` instead. `LANG/graph_utils.py` and `LANG/utils.py` files should only contain new methods and variables. Not all new languages will require a submodule specific `graph_utils.py` or `utils.py` file.
 
+2) NeMo-Text-Processing allows creation of FST graphs through two backends: the Python based library itself (via [Pynini](https://www.opengrm.org/twiki/bin/view/GRM/Pynini) backend) and C++ based [Sparrowhawk](https://github.com/google/sparrowhawk/tree/master) in an upstream repo. Due to the typical tradeoffs between these Python and C++ development [languages](https://www.youtube.com/watch?v=VioxsWYzoJk), the NeMo-Text-Processing library assumes development to be performed with the Python library for final deployment in Sparrowhawk/C++. This dual framework approach can lead to issues in development, notably in the case of tagging additional properties during tokenization.
+
+When writing taggers for semiotic classes, you may need to tag additional token properties (e.g. grammatical gender, case) for accurate verbalization. For example, the Spanish ordinal `21.º` carries masculine gender and is verbalized with a specific spelling. As such, it would be desired for the TN tagger to tokenize the string with the gender property included. 
+
+Naively, one may be tempted to simply include the property string `gender: "masc"` and check for this string during the verbalization phase. **This is not advised.** While the NeMo-Text-Processing library itself will permit any custom string in the tagger, Sparrowhawk limits permissible strings, and will fail with custom property strings. Given the performance loss in not providing Sparrowhawk support, we cannot integrate new graphs that cause Sparrowhawk failure. As such, tagged properties should be limited to Sparrowhawk supported strings. 
+
+For all classes, Sparrowhawk supports the `morphosyntactic_features` property, and it is recommended to default to this property for tagging additional features. For example:
+
+`21.º" -> ordinal { integer: "vigésimo primero" morphosyntactic_features: "masc" }`
+
+For additional Sparrowhawk supported properties by class, see [here](https://github.com/yzhang123/sparrowhawk/blob/test/src/proto/semiotic_classes.proto)
+
+N.B. The same limitation applies for novel semiotic classes as well. Only predefined classes are supported in Sparrowhawk.
+
+3) Between the tagging and verbalizing stages, both the NeMo-Text-Processing and Sparrowhawk engines permute order of tagged properties. That is, assuming the tagger parsed `1ᵉʳ juillet` as:
+
+`date { month: "juillet" day: "1" } }`
+
+the verbalizer will receive as input both
+
+`date { month: "juillet" day: "1" }`
+
+and
+
+`date { day: "1" month: "juillet" }`
+
+While this eases construction of verbalization graphs, permutation can be computationally expensive. If you know that the tagger output will not require permutation of token properties, you can improve model performance by including the `preserve_order: "true"` property:
+
+`date { day: "1" month: "juillet" preserve_order: "true" }`