Conversation
Signed-off-by: Kashif Rasul <kashif.rasul@gmail.com>
Signed-off-by: Kashif Rasul <kashif.rasul@gmail.com>
Signed-off-by: Kashif Rasul <kashif.rasul@gmail.com>
Signed-off-by: Kashif Rasul <kashif.rasul@gmail.com>
Signed-off-by: Kashif Rasul <kashif.rasul@gmail.com>
Signed-off-by: Kashif Rasul <kashif.rasul@gmail.com>
Signed-off-by: Kashif Rasul <kashif.rasul@gmail.com>
Signed-off-by: Kashif Rasul <kashif.rasul@gmail.com>
Signed-off-by: Kashif Rasul <kashif.rasul@gmail.com>
Signed-off-by: Kashif Rasul <kashif.rasul@gmail.com>
Signed-off-by: Kashif Rasul <kashif.rasul@gmail.com>
Signed-off-by: Kashif Rasul <kashif.rasul@gmail.com>
Signed-off-by: Kashif Rasul <kashif.rasul@gmail.com>
Signed-off-by: Kashif Rasul <kashif.rasul@gmail.com>
Signed-off-by: Kashif Rasul <kashif.rasul@gmail.com>
Signed-off-by: Kashif Rasul <kashif.rasul@gmail.com>
Signed-off-by: Kashif Rasul <kashif.rasul@gmail.com>
Signed-off-by: Kashif Rasul <kashif.rasul@gmail.com>
Signed-off-by: Kashif Rasul <kashif.rasul@gmail.com>
Signed-off-by: Kashif Rasul <kashif.rasul@gmail.com>
Signed-off-by: Kashif Rasul <kashif.rasul@gmail.com>
Signed-off-by: HuiyingLi <willwin.lee@gmail.com>
…ting Signed-off-by: HuiyingLi <willwin.lee@gmail.com>
The CP-free DSpark recipe shards the draft with fully_shard and no explicit mesh, so torch builds a one-dimensional default mesh whose mesh_dim_names is None. Exporting the draft's merged expert tensors to the HF layout then died in _get_expert_mesh_dim_index on tuple(None), which made every checkpoint save of a DeepSeek V4.1 DSpark run fail. Treat a missing name tuple as empty and, for an unnamed one-dimensional mesh, use the mesh itself as the expert partition instead of slicing it by name. Signed-off-by: khazic <khazzz1c@gmail.com>
c13a0e0 handled the unnamed mesh in get_expert_slice_for_rank, but split_experts_weights_dtensor_aware still listed mesh_dim_names on its own before rebuilding the per-expert DTensors, so the first checkpoint save of a DeepSeek V4.1 DSpark run still died with TypeError there. With no named dimension left after removing the expert partition, the per-expert slices are returned as plain tensors, the same outcome as a named single-dimension mesh. Signed-off-by: khazic <khazzz1c@gmail.com>
The released V4.1 DSpark confidence head reads the collapsed residual stream before the final norm, matching the official inference code, while the LM head reads the normalized states. Training this head with BCE therefore back-propagates through an unbounded activation: on some batches its logit jumps by tens, the confidence loss spikes by orders of magnitude while CE and TV stay flat, and that gradient degrades the draft backbone. Disabling the head removes the spikes but leaves it untrained, which serving needs. Add a recipe option that detaches the confidence head's inputs so its loss trains only confidence_head.proj. The served input and the exported weights are unchanged; the backbone receives the same gradient as with the head disabled. Off by default; the V4.1 example enables it. Covered by a unit test that checks where the confidence gradient lands with and without the flag. Signed-off-by: khazic <khazzz1c@gmail.com>
Signed-off-by: khazic <khazzz1c@gmail.com>
Pass the same named mesh to all draft FSDP units and preserve the CP data-parallel rank group. Restore shared MoE state-dict utilities to the PR base. Verify two-rank BF16 updates and safetensors checkpoint resume with and without activation checkpointing. Signed-off-by: HuiyingLi <willwin.lee@gmail.com>
Move text-only target policy and backend defaults into DeepseekV41DSparkTargetConfig.build. Keep distributed setup composition in the recipes and preserve their existing option names and overrides. Cover the model-owned loader contract and distributed precompute dispatch. Signed-off-by: HuiyingLi <willwin.lee@gmail.com>
Signed-off-by: khazic <khazzz1c@gmail.com>
The CPU suite reached 97-98% before the 20-minute timeout in all three attempts. Give the complete suite, including the DSpark distributed tests, enough time to finish. Individual test runtime budgets remain enforced. Signed-off-by: Huiying <willwin.lee@gmail.com>
Contributor
Author
|
/ok to test 7d5ffce |
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI-only proxy PR. Do not merge / do not review. Points at the exact head commit of #3875 (7d5ffce) so internal CI runs under the internal-contributor queue; results post back to #3875 via the shared SHA. Source of truth: #3875. Fixes are pushed to the original PR branch, then this proxy is synchronized. Close once CI completes.