Add transformers v5 compatibility #681
Conversation
mergify
bot
added
ci-failure
dependencies
📝 WalkthroughWalkthroughUpdate transformers dependency to >=5.0.0 and bump related pinned packages; replace tokenizer uses of Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Maxusmusti
force-pushed
the
transformer-v5-support
branch
from
0cba4d3 to
eb9b537
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
src/instructlab/training/data_process.py (1)
1301-1311: Update all test files to useextra_special_tokensinstead ofadditional_special_tokens.The codebase has a transformer v5 compatibility issue: production code in
data_process.pyandtokenizer_utils.pynow usesextra_special_tokens, but test files still use the deprecatedadditional_special_tokensAPI. This inconsistency will cause test failures.Update the following test files:
tests/unit/test_unmask_messages.py(lines 345, 495)tests/unit/test_data_process.py(lines 856, 912)
🤖 Fix all issues with AI agents
In `@src/instructlab/training/tokenizer_utils.py`:
- Around line 31-33: Reformat the tokenizer.add_special_tokens call to meet ruff
formatting (remove the extra line breaks/whitespace) so the call is a single
properly formatted expression; update the invocation of
tokenizer.add_special_tokens that references {"extra_special_tokens":
extra_special_tokens} to a single-line or otherwise ruff-compliant style (you
can also run `ruff format` on src/instructlab/training/tokenizer_utils.py) so
the linter/CI passes.
Maxusmusti
force-pushed
the
transformer-v5-support
branch
from
eb9b537 to
f2531d1
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In `@requirements.txt`:
- Line 6: The transformers>=5.0.0 bump may be incompatible with our code; verify
and either pin to a compatible version or update code/CI/docs: ensure Python
minimum is 3.10, add/update dependency constraints for huggingface-hub>=1.0.0
and tokenizers to avoid conflicts, and search for and replace removed/deprecated
APIs—e.g., replace load_in_4bit/load_in_8bit usage with quantization_config,
change model loading dtype handling (auto vs fp32), replace use_auth_token with
token, remove references to transformers-cli, update Trainer usages
(renamed/removed args), and adapt tokenizers API calls—update tests, CI images,
and docs accordingly so all occurrences (load_in_4bit, load_in_8bit,
quantization_config, use_auth_token, Trainer, transformers-cli, tokenizers.*)
are fixed or dependency pinned.
| py-cpuinfo | ||
| torch>=2.6.0 | ||
| transformers>=4.55.0 | ||
| transformers>=5.0.0 |
There was a problem hiding this comment.
🧩 Analysis chain
🌐 Web query:
transformers 5.0.0 minimum Python version requirements dependencies
💡 Result:
transformers 5.0.0 minimum Python version
- Python >= 3.10.0 (this is the package’s
Requires-Pythonon PyPI). [1]
Note: the project description on PyPI also says “Python 3.9+”, but the install metadata for 5.0.0 is stricter (
>=3.10.0). [1]
Runtime (core) dependencies (required)
Listed dependencies for transformers on piwheels are: filelock, huggingface-hub, numpy, packaging, pyyaml, regex, requests, safetensors, tokenizers, tqdm. [2]
Framework requirement (practical)
- The PyPI description for 5.0.0 states it works with PyTorch 2.1+ (installed separately; commonly via
pip install "transformers[torch]"). [1]
Sources: [1], [2]
🌐 Web query:
transformers 5.0.0 breaking changes release notes
💡 Result:
Here are the Transformers 5.0.0 breaking / backward-incompatible changes, straight from the official v5.0.0 release notes (released Jan 26, 2026). [1]
Library-wide
from_pretrained(... )default dtype is nowauto(instead of forcing FP32). [2]- Default Hub shard size is now 50GB (up from 5GB). [2]
use_auth_tokenis deprecated in favor oftokeneverywhere. [2]
Attention / Torch / Quantization
- Removed: head masking, relative positional biases in BERT-like models, and head pruning. [3]
- Dropped support for
torchscriptandtorch.fx(focus is now ondynamo/export). [3] - Removed quantization args
load_in_4bitandload_in_8bit→ usequantization_configinstead. [4]
Configuration
- Removed nested-config init helpers like
from_xxx_config. [4] - Can’t load a config class from a URL anymore (must be local path or Hub repo). [4]
- RoPE config changes: parameters moved under
config.rope_parameters; e.g. accessingconfig.rope_thetacan now error. [4] - Non-generative models no longer have
generation_config;model.config.generation_configmay now error. [5]
Tokenization / Processing
- Tokenizers: consolidation away from slow/fast split; single
tokenization_<model>.pywith backend selection. [6] - Tokenization breaking changes include (highlights): legacy tokenizer files removal,
encode_plus -> __call__,batch_decode -> decode, andapply_chat_templatenow returns aBatchEncoding. [5] - Processing: processor serialization changes;
XXXFeatureExtractorsremoved in favor ofXXXImageProcessorfor vision models. [5]
Modeling / Generate / Trainer / Hub / CLI
- All models’
forward()now accept**kwargs. [7] - Generate: removed deprecated output aliases and some deprecated decoding-related classes. [7]
- Trainer/TrainingArguments: many deprecated args removed/renamed (e.g.
no_cuda -> use_cpu,per_gpu_* -> per_device_*, etc.). [8] - PushToHubMixin: removed/changed args;
push_to_hubis now keyword-only (exceptrepo_id), no**kwargs, removeduse_temp_dir, etc. [8] - CLI:
transformers-cliremoved;transformers ...is the only entry point;transformers chat/servesplit, andtransformers runremoved. [8] - Env vars: removed legacy cache env vars like
TRANSFORMERS_CACHEin favor ofHF_HOME; plushuggingface_hub-related changes (e.g. requests→httpx error type). [9]
If you tell me what you’re upgrading from (e.g. 4.57.x → 5.0.0) and which parts you use (Trainer, tokenizers, quantization, CLI), I can produce a short “search/replace + code changes” checklist.
🌐 Web query:
transformers 5.0.0 PyPI package metadata tokenizers huggingface-hub compatibility
💡 Result:
For transformers==5.0.0 (released Jan 26, 2026), the published package metadata indicates:
- Python requirement:
Python >=3.10(PyPI) [1] - Core runtime deps include:
huggingface-hubandtokenizers(among others likenumpy,safetensors,tqdm, etc.) [2] - Compatibility change in v5:
transformersnow requireshuggingface_hub >= 1.0.0[3]
What this means for tokenizers / huggingface-hub compatibility
Because transformers depends on both huggingface-hub and tokenizers [2] and also enforces huggingface_hub >= 1.0.0 [3], you must use a tokenizers version that does not pin huggingface_hub to <1.0.0—otherwise pip’s resolver will fail with a version conflict (this kind of conflict is documented historically when tokenizers pinned huggingface_hub to an upper bound) [4].
Practical rule: with transformers==5.0.0, keep huggingface_hub on 1.x and ensure tokenizers is new enough to be compatible with that.
Sources:
[1] https://pypi.org/project/transformers/
[2] https://www.piwheels.org/project/transformers/
[3] https://newreleases.io/project/pypi/transformers/release/5.0.0
[4] huggingface/transformers#26276
Verify Transformers v5.0.0 compatibility—breaking changes affect multiple APIs.
The bump to >=5.0.0 introduces major breaking changes beyond Python/dependency updates:
- Python: Minimum is now 3.10 (not 3.9).
- Dependencies: Requires
huggingface-hub >= 1.0.0andtokenizers(ensure no version conflicts; oldertokenizersmay pinhuggingface-hubto <1.0.0). - Breaking changes: Model loading dtype now defaults to
auto(not FP32); quantization args (load_in_4bit,load_in_8bit) removed in favor ofquantization_config; tokenizers API consolidated; Trainer args renamed/removed;transformers-cliremoved entirely;use_auth_tokendeprecated fortoken; and many more.
Confirm your code does not use deprecated/removed APIs (quantization, trainer args, CLI, tokenizers functions, model config patterns). Update all affected code, CI, and docs accordingly.
🤖 Prompt for AI Agents
In `@requirements.txt` at line 6, The transformers>=5.0.0 bump may be incompatible
with our code; verify and either pin to a compatible version or update
code/CI/docs: ensure Python minimum is 3.10, add/update dependency constraints
for huggingface-hub>=1.0.0 and tokenizers to avoid conflicts, and search for and
replace removed/deprecated APIs—e.g., replace load_in_4bit/load_in_8bit usage
with quantization_config, change model loading dtype handling (auto vs fp32),
replace use_auth_token with token, remove references to transformers-cli, update
Trainer usages (renamed/removed args), and adapt tokenizers API calls—update
tests, CI images, and docs accordingly so all occurrences (load_in_4bit,
load_in_8bit, quantization_config, use_auth_token, Trainer, transformers-cli,
tokenizers.*) are fixed or dependency pinned.
| tokenizer.add_special_tokens( | ||
| {"additional_special_tokens": additional_special_tokens} | ||
| ) | ||
| extra_special_tokens = tokenizer.extra_special_tokens + new_tokens |
There was a problem hiding this comment.
Lol these tokens are extra special
RobotSail
left a comment
RobotSail
left a comment
There was a problem hiding this comment.
LGTM. Looks like there are some unit tests failing still, once those are fixed this can merge
Updates the library to support HuggingFace Transformers v5.
Changes
additional_special_tokenswithextra_special_tokensin tokenizer API calls (renamed in v5)>=5.0.0Files Modified
requirements.txtsrc/instructlab/training/tokenizer_utils.pysrc/instructlab/training/data_process.pysrc/instructlab/training/main_ds.pySummary by CodeRabbit
Chores
Improvements
✏️ Tip: You can customize this high-level summary in your review settings.