Add public SentencePieceTokenizer factory methods for Unigram from vocab list and tokenizer.json#7625

Open

Copilot wants to merge 15 commits into

mainfrom

copilot/add-sentencepiece-tokenizer-api

Commits on Jun 10, 2026

Commits on Jun 11, 2026

Fix BOS/EOS positional fallback, normalizer type validation, and prepend_scheme handling

Copilot
and
ericstj
authored
Fix null normalizer guard, recursive Sequence support, BOS/EOS validation, and add tests

Copilot
and
ericstj
authored

Commits on Jun 12, 2026

Generalize SentencePiece special tokens to prefix/suffix lists with post-processor support

ericstj
and
Copilot
committed

Commits on Jun 18, 2026