Skip to content

fix(ci): match Nemotron remote reloads with legacy PEFT exports - #3952

Merged
yuhezhang-ai merged 6 commits into
mainfrom
yuhez/fix/amint-356-hf-peft-reload
Sep 22, 2026
Merged

yuhezhang-ai merged 6 commits into
mainfrom
yuhez/fix/amint-356-hf-peft-reload

Conversation

@yuhezhang-ai

@yuhezhang-ai yuhezhang-ai commented Sep 20, 2026

Copy link
Copy Markdown
Contributor

What does this PR do ?

Nemotron PEFT checkpoints should work with either HF's newer built-in model or the legacy remote-code model, using the matching export format:

  • Newer built-in HF model: use AutoModel's default checkpoint export.
  • Legacy HF remote-code model: use checkpoint.v4_compatible: true with trust_remote_code: true.

We are using the legacy path in CI for now because we found a bug in HF's built-in Nemotron Mamba implementation that causes numerical mismatches. We submitted the fix upstream as huggingface/transformers#48989. The current CI results validate the legacy path; they do not establish a full built-in-model CI pass.

This PR switches four checkpoint-robustness tests to the matching legacy export, keeping our existing HF dependency rather than pinning an unreleased patched build. The previous tests mixed the default export's model.* adapter names with a remote-code model expecting legacy backbone.* names, causing missing-adapter failures.

The export override lives under ci.checkpoint_robustness, so it applies only to checkpoint tests. Normal recipe export settings remain unchanged.

Stacked on #3953 for its fixed Mamba cumsum parity configuration. The diff against that base adds only the legacy-export override and explanatory comment to each recipe.

Changelog

  • Enable the test-only legacy PEFT export for Customizer Nano, Customizer packing, Nano 4B SQuAD and Nano v3 HellaSwag.
  • Preserve remote-code HF reloads, all six checkpoint phases, exact adapter-weight checks and numerical thresholds.

Before your PR is "Ready for review"

  • DCO signoff and contributor guidance followed.
  • Ruff format/check and Git diff checks pass.
  • All four real harness/parser checks pass: legacy export is enabled only for checkpoint robustness; other recipe settings match the base branch.
  • 46 focused Mamba and state-dict adapter unit tests pass.
  • Generated CI retains all six phases for all four recipes.
  • Scoped EOS CI: pipeline 69055637 and recipe pipeline 69056142 pass at pushed commit 446557db8d91b24dd920bb408fae6b68e19dce00.

All four recipes pass all six phases: source reference, source parity, training/save, AutoModel reload, HF reload and resume. AutoModel reload outputs are bitwise identical in all four; HF reload verifies every saved adapter tensor exactly and passes the existing numerical thresholds.

Recipe CI job Adapter tensors verified HF reload mean KL
Customizer Nano Pass 188 0.002814
Customizer Nano packing Pass 188 0.002868
Nano 4B SQuAD Pass 144 0.000507
Nano v3 HellaSwag Pass 188 0.002477

Additional Information

Addresses the legacy-export/receiver mismatch in AMINT-356. The upstream HF fix includes regression tests and controlled numerical evidence. Once a suitable HF release includes the fix, the built-in path can be validated again for these CI tests.

@copy-pr-bot

copy-pr-bot Bot commented Sep 20, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>
Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>
Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>
Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>
@yuhezhang-ai
yuhezhang-ai force-pushed the yuhez/fix/amint-356-hf-peft-reload branch from 334a8c7 to 7156f57 Compare September 21, 2026 01:11
@yuhezhang-ai
yuhezhang-ai changed the base branch from main to yuhez/investigate/gptoss-length-parity September 21, 2026 01:11
Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>
@yuhezhang-ai yuhezhang-ai changed the title fix(ci): reload Nemotron PEFT adapters with built-in HF models fix(ci): match Nemotron remote reloads with legacy PEFT exports Sep 21, 2026
@yuhezhang-ai yuhezhang-ai added the docs-only With great power comes great responsibility. label Sep 21, 2026
Base automatically changed from yuhez/investigate/gptoss-length-parity to main September 21, 2026 17:58
@yuhezhang-ai

Copy link
Copy Markdown
Contributor Author

/ok to test dd2b963

@yuhezhang-ai
yuhezhang-ai marked this pull request as ready for review September 21, 2026 19:39
@yuhezhang-ai
yuhezhang-ai requested a review from a team as a code owner September 21, 2026 19:39
@yuhezhang-ai
yuhezhang-ai merged commit 9c7d130 into main Sep 22, 2026
39 checks passed
@yuhezhang-ai
yuhezhang-ai deleted the yuhez/fix/amint-356-hf-peft-reload branch September 22, 2026 17:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs-only With great power comes great responsibility.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants