Report the unscaled gradient norm from FP16_UnfusedOptimizer - #8588
vineethsaivs wants to merge 1 commit into
Conversation
Keep the scaled norm in a local for clipping and stash the unscaled one, matching the fused optimizer. The clip decision is unchanged. Signed-off-by: Vineeth Sai <vineethsai4444@gmail.com>
|
Could you approve the CI run? Two regressions fail before, reporting 128.0 for a true norm of 1.0 at a static loss scale of 128, on both step paths. They pass after, and pin that clipping still gets the scaled norm. Changed-file pre-commit passes. |
|
One adjacent thing I found while checking the siblings, deliberately left out of this diff. The two classes also disagree about MoE. That is a behaviour change to clipping rather than to reporting, which is what this PR is about, and this PR states the clip decision is unchanged. Happy to open it separately, or fold it in here if you would rather have both at once. |
DeepSpeedEngine.get_global_grad_norm()is documented as the 2-norm of all gradients, but on the fp16 non-fused path it returns that norm still multiplied by the loss scale.stepandstep_fused_lambstash the norm built from the fp16 gradients, which still carry the loss scale. The fused optimizer divides it out, and this class's ownunscale_and_clip_gradsdoes too. Keep the scaled norm in a local for clipping and stash the unscaled one, matching the fused optimizer. The clip decision is unchanged.Test:
TORCHDYNAMO_DISABLE=1 DS_ACCELERATOR=cpu PYTHONPATH=. python -m pytest tests/unit/runtime/half_precision/test_unfused_optimizer.py -q. Two regressions fail before, reporting 128.0 for a true norm of 1.0 at a static loss scale of 128, on both step paths. They pass after, and pin that clipping still gets the scaled norm. Changed-file pre-commit passes. Executed on Apple M2 Pro CPU; no CUDA or distributed training run.