-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Pull requests: huggingface/tokenizers
Author
Label
Projects
Milestones
Reviews
Assignee
Sort
Pull requests list
feat(python): add Arrow string batch tokenization
#2400
opened Sep 10, 2026 by
granthamtaylor
Loading…
bpe: do not slice merge tokens at a byte offset that may not be a boundary
#2398
opened Sep 10, 2026 by
apollo-2006
Loading…
chore(deps): bump js-yaml from 4.3.1 to 4.3.2 in /bindings/node
dependencies
Pull requests that update a dependency file
javascript
Pull requests that update Javascript code
#2397
opened Sep 10, 2026 by
dependabot
Bot
Loading…
Fix BERT normalization of CJK Extension E boundary characters
#2395
opened Sep 8, 2026 by
mottopanikeiku
Loading…
fix: Unigram prune_sentence_pieces uses alternatives[id].len() not alternatives.len()
#2394
opened Sep 8, 2026 by
VirajMishra1
Loading…
fix(python): forward unk_id in SentencePieceUnigramTokenizer
#2393
opened Sep 7, 2026 by
Con-Benksl
Loading…
fix(bpe): drop heap entries with non-positive pair_counts before u64 cast
#2389
opened Sep 3, 2026 by
okxint
Loading…
perf(decode): index the vocab slab by id, not through the MPHF slot
#2387
opened Sep 3, 2026 by
ArthurZucker
Collaborator
Loading…
fix: correctly decode numpy UCS-4 arrays instead of treating as UTF-8
#2386
opened Sep 3, 2026 by
VirajMishra1
Loading…
docs: rewrite the README for the crate split
#2385
opened Sep 2, 2026 by
ArthurZucker
Collaborator
Loading…
v1.0.0-rc0: the train/encode split, repacked into 53 commits
#2384
opened Sep 2, 2026 by
ArthurZucker
Collaborator
Loading…
perf(normalizers): fast-path BertNormalizer for non-CJK input
#2383
opened Sep 2, 2026 by
bsachart
Loading…
Fix UnicodeScripts pre-tokenizer dropping leading spaces
#2382
opened Sep 2, 2026 by
Kayvan-Zahiri
Loading…
perf(parallelism): default the pool to physical cores, not SMT siblings
#2380
opened Sep 2, 2026 by
ArthurZucker
Collaborator
Loading…
perf(train): select top-k in the trainers instead of ordering everything
#2379
opened Sep 2, 2026 by
ArthurZucker
Collaborator
Loading…
perf(word-cache): scan the tag window with shrn, not an emulated movemask
#2377
opened Sep 2, 2026 by
ArthurZucker
Collaborator
Loading…
perf(pre_tokenizers): eliminate per-token heap allocations in ByteLevel decoder
#2376
opened Sep 1, 2026 by
bsachart
Loading…
feat: support
first prepend scheme for metaspace
#2372
opened Aug 31, 2026 by
SBrandeis
Contributor
Loading…
Fix BPE training across consecutive line endings
#2370
opened Aug 30, 2026 by
Alphaxiaoteng
Loading…
fix(python): preserve async encode character offsets
#2369
opened Aug 28, 2026 by
gry67673905
Loading…
Add a
rusty-expressions regex backend (Oniguruma semantics, pure Rust, wasm-capable)
#2368
opened Aug 27, 2026 by
Ttimmahlax
Loading…
Previous Next
ProTip!
Adding no:label will show everything without a label.