Bilingual word alignment using multilingual embeddings — no training required!
For non-nerds: a word matching engine for any language pair using parallel data.
pip install vectoralignfrom vectoralign import align
# Your parallel sentences
english = [
"Hello world",
"How are you today",
"The weather is nice"
]
hindi = [
"नमस्ते दुनिया",
"आज आप कैसे हैं",
"मौसम अच्छा है"
]
# Align and build dictionary
dictionary = align(english, hindi)- No training required — Uses pre-trained multilingual embeddings
- Batch processing — Batch processing with automatic CUDA detection
- Multiple models — Supports LaBSE, mBERT, and other HuggingFace models
- Bidirectional alignment — Intersection of forward and backward alignments
- Memory management — Automatic garbage collection and CUDA cache management for efficient memory usage
from vectoralign import align
# Custom model and batch size
dictionary = align(
src_sentences,
tgt_sentences,
model_name="setu4993/LaBSE", # or "bert" for mBERT
batch_size=64,
threshold=0.6, # Sentence similarity threshold
output="my_dictionary.txt"
)| Model | Name |
|---|---|
| LaBSE | setu4993/LaBSE (default) |
| mBERT | bert with mode='multilingual' |
Any HuggingFace model with pooler_output |
Full model path |
The dictionary is saved as a TSV file:
word1 translation1 count
word2 translation2 count
This is a spiritual implementation and improvement of SimAlign by the Centre for Language and Information Processing, LMU Munich.
MIT License