Repository navigation
Conversation
…ters ZeRO-1/2 and BF16_Optimizer copy each rank's fp32 master partition into its bf16 parameters with round-to-nearest after every optimizer step. A master update smaller than half a bf16 spacing does not change the bf16 parameter, so at small learning rates the parameters used by the forward pass and saved in weights-only exports trail the masters by many steps. Add bf16.stochastic_round_params (default false). When set, that copy rounds each element to the bf16 neighbour away from zero with probability equal to its fractional position between its two bf16 neighbours, so the bf16 parameters equal the masters in expectation. The rounding happens before the existing all-gather, so it adds no communication. Draws are seeded from the torch seed and the data-parallel rank, so tensor-parallel replicas of a parameter round identically. The engine refuses the key for optimizers that do not apply it (ZeRO-3, ZenFlow, FP16_Optimizer), and the config refuses it with bf16_master_weights_and_grads. Test: two CPU ranks, eight Adam steps on a constant gradient that move every master less than half a spacing. With the key off, the bf16 parameters stay at round-to-nearest of the masters (unchanged); with it on, they are unbiased within 5 standard errors, for BF16_Optimizer and ZeRO-2 with and without CPU offload. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
bf16.stochastic_round_params(defaultfalse). When set, ZeRO-1/2 andBF16_Optimizercopy the fp32 master weights into the bf16 parameters with stochastic rounding instead of round-to-nearest after every optimizer step, so the bf16 parameters equal the masters in expectation and updates smaller than half a bf16 spacing still reach them.This is a new feature, and DeepSpeed's new-feature process starts with a proposal issue. None is open yet. Motivation and planned experiments are below; the submitter will move them to an issue if maintainers prefer to discuss there.
Problem
After each step,
DeepSpeedZeroOptimizer.step()(stage_1_and_2.py, both the offload and non-offload branches) andBF16_Optimizer.update_lp_params()copy each rank's fp32 master partition into its bf16 partition withcopy_, which rounds to nearest, and then all-gather the bf16 parameters. A master update smaller than half the bf16 spacing (the spacing nearwis between|w|/256and|w|/128) leaves the bf16 parameter unchanged. The update waits in the master until it crosses a rounding midpoint, and until then the parameters used for forward/backward and by anything reading the module trail the masters. With Adam atlr = 1e-6, one step cannot move the bf16 value of any element with|w| > 5.1e-4. Example: at|w| = 0.01the bf16 spacing is2^-14 = 6.1e-5, so an element starting on the bf16 grid needs 31 consecutive same-sign steps atlr = 1e-6before its bf16 value moves at all.Gao and Gao, "bf16 Rounding Silently Discards Most RL Policy Updates: A Hidden Training-Inference Mismatch and an Unbiased One-Line Fix" (arXiv:2610.41073), describe this for RL post-training at small learning rates and propose rounding the master-to-bf16 copy stochastically ("Fair Cast"). On Qwen3 0.6B to 4B with AdamW and Muon at lr 1e-6, they report it ahead of round-to-nearest by more than 1 point on 18 of 25 matched seed pairs, with a mean of +6.1 held-out points. Those are the paper's results; they have not been reproduced here.
Change
deepspeed/runtime/stochastic_rounding.py(new, 52 lines):stochastic_round_to_bf16_(dst, src, generator)adds 16 uniform random bits below bf16's last mantissa bit to the fp32 bit pattern and truncates them. Because fp32 is sign-magnitude, this rounds the magnitude up with probability equal to the dropped fraction, for either sign and for subnormals. Non-finite values round to nearest. It works in chunks of 2^24 elements.StochasticRoundingCopyis acopy_(dst, src)replacement holding onetorch.Generatorper device, seeded fromtorch.initial_seed()and the optimizer's data-parallel rank. Data-parallel ranks therefore draw different bits, while tensor-parallel ranks with the same torch seed hold a replicated parameter (a LayerNorm weight, say) at the same offsets and round it identically, as round-to-nearest does. Destinations that are not bf16 (parameters kept in fp32) are copied exactly.DeepSpeedZeroOptimizerandBF16_Optimizerpick their master-to-bf16 copy once at construction,torch.Tensor.copy_(unchanged default) orStochasticRoundingCopy, and call it at the existing copy sites: both branches of the ZeRO-1/2 step, ZeRO-1/2update_lp_params()(universal-checkpoint load) andBF16_Optimizer.update_lp_params(). No new branches in the step.DeepSpeedBF16Config.stochastic_round_params, refused together withbf16_master_weights_and_grads(there is no fp32 master to round from). The engine passes it to ZeRO-1/2 and refuses it when the optimizer wrapper is anything other than exactlyBF16_OptimizerorDeepSpeedZeroOptimizer(ZeRO-3, ZenFlow,FP16_Optimizer), where it would otherwise be silently ignored.docs/_pages/config-json.md: the new key.Production code: 52 new lines in the new module and 37 added / 4 removed lines in four existing files (
engine.py+10,precision_config.py+14,stage_1_and_2.py+8/-3,bf16_optimizer.py+5/-1).Tradeoffs and compatibility
torch.Tensor.copy_through an attribute.randintand a few elementwise operations per master element per step, with about 150 MB of temporaries per 2^24-element chunk, on the device holding the master partition. Only the rank owning a partition rounds it, before the existing all-gather, so there is no extra communication and no host synchronization. Not measured on an accelerator. With CPU optimizer offload the rounding runs on the host, where the draw is serial (MT19937): 7.7 ns per element on 64 threads of a Xeon Platinum 8375C, of whichrandintis 5.2, against 0.02 ns for the round-to-nearestcopy_. That is about 8 s per step for 10^9 master elements per rank. Drawing on the accelerator instead would cost a 4-byte instead of 2-byte host-to-device copy per element; this PR leaves that for later and documents the cost.update_lp_params), so they are a stochastic rounding of the loaded masters; withload_module_onlythe followingrefresh_fp32_paramsthen copies that rounding back into the masters, as it copies the round-to-nearest value today.stage3.py_reassign_or_swap_out_partitioned_parametersand its pinned-master and NVMe swap copies),FP16_Optimizerwith bf16, and ZenFlow's ownstep()copy also round to nearest. All three are refused rather than changed; ZeRO-3 could follow.torch.Generator(device=...)andtorch.randint(generator=...)on the master's device. Exercised only on CPU here; CUDA and other accelerators are untested.bf16.store_param_remainders) assumes the bf16 parameter is the master rounded to nearest, so the two options are mutually exclusive. Whichever lands second should refuse the combination.Validation
Hardware and software: CPU only. Intel Xeon Platinum 8375C (x86_64), no accelerator;
DS_ACCELERATOR=cpu, gloo backend, 2 processes; Python 3.12.3, torch 2.11.0+cu130 (running on CPU), pydantic 2.13.3, pytest 9.0.3.Result: 8 passed. The whole file,
unit/v1/half_precision/test_bf16.py: 35 passed, 6 skipped.TestBF16StochasticRoundParams(2 ranks)BF16_Optimizer(ZeRO-1, fp32 grad accumulation); ZeRO-2; ZeRO-2 + CPU optimizer offload; each with the key off and on|w|in [0.3, 1), constant gradient, 8 Adam steps at lr 1e-4: every master moves by 8e-4 (checked against the analytic value to 1e-6), less than half a spacing. Off: the bf16 parameters equal the masters rounded to nearest and have not moved. On: every element is one of its master's two bf16 neighbours (fromtorch.nextafter), the summed rounding error is within 5 standard errors of zero, and all ranks hold identical parameters.TestBF16StochasticRoundParamsUnsupported(1 rank)bf16_master_weights_and_gradsdeepspeed.initializeraisesValueErrornaming the key instead of silently ignoring it.Red checks, each run on a copy of this branch with one part reverted:
copy_)initializereturns, and the worker then did not exit before the 7- and 20-minute timeouts. It never passes, but it does not fail cleanly either.bf16_master_weights_and_gradscase fails.pre-commit run --files <changed files>: all hooks pass.Not validated:
immediate_grad_update,torch_autocast, Muon and universal-checkpoint load with the key on. Each reaches the same copy sites, by source reading only.Planned experiments
The option changes convergence, so per CONTRIBUTING it needs full-training comparisons against round-to-nearest. Proposed, for maintainers to adjust:
Open question for maintainers: should the first version round on the accelerator under CPU offload (double the host-to-device bytes on that path) instead of paying the serial host draw?
AI assistance
This change, its tests and this description were written with AI assistance (Claude). The human submitter reviews every line, runs the device tests and signs off before this leaves draft.