mem: load only 15 common langdetect profiles to reduce memory

[KRRT7]

Monkey-patch langdetect's init_factory() to load only 15 high-coverage language profiles instead of all 55. The full set creates large n-gram probability arrays (word_lang_prob_map) that persist for the lifetime of the process. The subset covers >95% of web content by language share.

What happens with unsupported languages?

langdetect always returns a result — it scores text against loaded profiles using n-gram frequency comparison and picks the best match. For a document in an excluded language (e.g., Thai, Polish), langdetect returns the closest loaded profile instead of the correct one. This is the same behavior as before for the downstream code path: detect_languages() maps the langdetect result to an ISO 639-3 code for element metadata. The language metadata is informational — it does not gate OCR model selection or partitioning logic.

In practice this means:

• Included language (e.g., English, French, Japanese): correct detection, same as before
• Excluded language (e.g., Thai): langdetect returns the closest loaded profile (e.g., id for Thai text) instead of th. No crash, no empty result — just a less precise language tag in metadata.

The 15 languages were chosen to cover the vast majority of real-world document processing: en, es, ar, fr, de, it, pt, ru, ja, ko, zh-cn, zh-tw, hi, bn, id. If a deployment needs accurate metadata for additional languages, the set can be extended by adding to LANGDETECT_LANGUAGES in lang.py.

Benchmark

Measured with memray (memray run + memray stats --json), 3 rounds × 5 texts (EN/DE/FR/JP + short ASCII) through the real detect_languages() code path, Python 3.12.

detect_languages() — langdetect profile subsetting benchmark
unstructured.partition.common.lang  |  3 rounds x 5 texts  |  Python 3.12.12

Configuration                     Peak MB      Saved      %
------------------------------------------------------------
All 55 profiles                    76.1MB      0.0MB   0.0%
15 common profiles                 31.7MB     44.4MB  58.3%