Skip to content

bf16: opt-in stochastic rounding of the bf16 parameters from fp32 masters - #9136

Draft
seanbell wants to merge 1 commit into
deepspeedai:masterfrom
seanbell:bf16-stochastic-round-params
Draft

seanbell wants to merge 1 commit into
deepspeedai:masterfrom
seanbell:bf16-stochastic-round-params

Conversation

@seanbell

Copy link
Copy Markdown
Contributor

Adds bf16.stochastic_round_params (default false). When set, ZeRO-1/2 and BF16_Optimizer copy the fp32 master weights into the bf16 parameters with stochastic rounding instead of round-to-nearest after every optimizer step, so the bf16 parameters equal the masters in expectation and updates smaller than half a bf16 spacing still reach them.

This is a new feature, and DeepSpeed's new-feature process starts with a proposal issue. None is open yet. Motivation and planned experiments are below; the submitter will move them to an issue if maintainers prefer to discuss there.

Draft. Not yet run on any accelerator (no GPU was available to the author), no convergence, throughput or memory results yet, and the commit carries no Signed-off-by. All three wait for the human submitter, who reviews every changed line, runs the device tests and signs off. "Validation" lists exactly what was and was not run. Maintainers are welcome to push to this branch or take the change over.

Problem

After each step, DeepSpeedZeroOptimizer.step() (stage_1_and_2.py, both the offload and non-offload branches) and BF16_Optimizer.update_lp_params() copy each rank's fp32 master partition into its bf16 partition with copy_, which rounds to nearest, and then all-gather the bf16 parameters. A master update smaller than half the bf16 spacing (the spacing near w is between |w|/256 and |w|/128) leaves the bf16 parameter unchanged. The update waits in the master until it crosses a rounding midpoint, and until then the parameters used for forward/backward and by anything reading the module trail the masters. With Adam at lr = 1e-6, one step cannot move the bf16 value of any element with |w| > 5.1e-4. Example: at |w| = 0.01 the bf16 spacing is 2^-14 = 6.1e-5, so an element starting on the bf16 grid needs 31 consecutive same-sign steps at lr = 1e-6 before its bf16 value moves at all.

Gao and Gao, "bf16 Rounding Silently Discards Most RL Policy Updates: A Hidden Training-Inference Mismatch and an Unbiased One-Line Fix" (arXiv:2610.41073), describe this for RL post-training at small learning rates and propose rounding the master-to-bf16 copy stochastically ("Fair Cast"). On Qwen3 0.6B to 4B with AdamW and Muon at lr 1e-6, they report it ahead of round-to-nearest by more than 1 point on 18 of 25 matched seed pairs, with a mean of +6.1 held-out points. Those are the paper's results; they have not been reproduced here.

Change

  • deepspeed/runtime/stochastic_rounding.py (new, 52 lines): stochastic_round_to_bf16_(dst, src, generator) adds 16 uniform random bits below bf16's last mantissa bit to the fp32 bit pattern and truncates them. Because fp32 is sign-magnitude, this rounds the magnitude up with probability equal to the dropped fraction, for either sign and for subnormals. Non-finite values round to nearest. It works in chunks of 2^24 elements. StochasticRoundingCopy is a copy_(dst, src) replacement holding one torch.Generator per device, seeded from torch.initial_seed() and the optimizer's data-parallel rank. Data-parallel ranks therefore draw different bits, while tensor-parallel ranks with the same torch seed hold a replicated parameter (a LayerNorm weight, say) at the same offsets and round it identically, as round-to-nearest does. Destinations that are not bf16 (parameters kept in fp32) are copied exactly.
  • DeepSpeedZeroOptimizer and BF16_Optimizer pick their master-to-bf16 copy once at construction, torch.Tensor.copy_ (unchanged default) or StochasticRoundingCopy, and call it at the existing copy sites: both branches of the ZeRO-1/2 step, ZeRO-1/2 update_lp_params() (universal-checkpoint load) and BF16_Optimizer.update_lp_params(). No new branches in the step.
  • DeepSpeedBF16Config.stochastic_round_params, refused together with bf16_master_weights_and_grads (there is no fp32 master to round from). The engine passes it to ZeRO-1/2 and refuses it when the optimizer wrapper is anything other than exactly BF16_Optimizer or DeepSpeedZeroOptimizer (ZeRO-3, ZenFlow, FP16_Optimizer), where it would otherwise be silently ignored.
  • docs/_pages/config-json.md: the new key.

Production code: 52 new lines in the new module and 37 added / 4 removed lines in four existing files (engine.py +10, precision_config.py +14, stage_1_and_2.py +8/-3, bf16_optimizer.py +5/-1).

Tradeoffs and compatibility

  • Default path: unchanged. The only difference is calling torch.Tensor.copy_ through an attribute.
  • Cost when enabled: one 16-bit randint and a few elementwise operations per master element per step, with about 150 MB of temporaries per 2^24-element chunk, on the device holding the master partition. Only the rank owning a partition rounds it, before the existing all-gather, so there is no extra communication and no host synchronization. Not measured on an accelerator. With CPU optimizer offload the rounding runs on the host, where the draw is serial (MT19937): 7.7 ns per element on 64 threads of a Xeon Platinum 8375C, of which randint is 5.2, against 0.02 ns for the round-to-nearest copy_. That is about 8 s per step for 10^9 master elements per rank. Drawing on the accelerator instead would cost a 4-byte instead of 2-byte host-to-device copy per element; this PR leaves that for later and documents the cost.
  • Rounding noise: each step redraws every element's rounding, so elements whose master barely moved can still flip by one bf16 spacing between steps. That is the price of unbiasedness; the paper's method redraws every step too.
  • Determinism and checkpoints: draws come from a dedicated generator, so the user's RNG streams (dropout etc.) are untouched. The generator state is not checkpointed: a restarted run re-seeds and replays the same draw sequence from its first step, which stays unbiased. Nothing is added to checkpoints and no format changes. A universal-checkpoint load also refreshes the bf16 parameters through the same copy (update_lp_params), so they are a stochastic rounding of the loaded masters; with load_module_only the following refresh_fp32_params then copies that rounding back into the masters, as it copies the round-to-nearest value today.
  • Pipeline-tied parameters (an embedding shared by the first and last stage) have equal masters but sit at different offsets on each stage, so they can round differently, where round-to-nearest keeps them bitwise equal.
  • Skipped steps: an overflow-skipped step returns before the copy, so it draws nothing.
  • Sibling paths checked: ZeRO-3 (stage3.py _reassign_or_swap_out_partitioned_parameters and its pinned-master and NVMe swap copies), FP16_Optimizer with bf16, and ZenFlow's own step() copy also round to nearest. All three are refused rather than changed; ZeRO-3 could follow.
  • Accelerators: uses torch.Generator(device=...) and torch.randint(generator=...) on the master's device. Exercised only on CPU here; CUDA and other accelerators are untested.
  • Related open PR: ZeRO-1/2: store master weights as the bf16 parameter plus a 16-bit remainder #9108 (bf16.store_param_remainders) assumes the bf16 parameter is the master rounded to nearest, so the two options are mutually exclusive. Whichever lands second should refuse the combination.
  • Not a duplicate: searched open and closed PRs and issues for "stochastic rounding", "stochastic_rounding", "stochastic round bf16", "round-to-nearest bf16 master" and "fair cast". The only stochastic-rounding work is Add subchannel FP8/INT8 quantized training for nn.Linear and AutoEP experts #9131/[RFC] Subchannel FP8/INT8 quantized training for nn.Linear and AutoEP experts (Triton, CUDA) #9132, which rounds quantized GEMM operands, not the master-to-parameter copy.

Validation

Hardware and software: CPU only. Intel Xeon Platinum 8375C (x86_64), no accelerator; DS_ACCELERATOR=cpu, gloo backend, 2 processes; Python 3.12.3, torch 2.11.0+cu130 (running on CPU), pydantic 2.13.3, pytest 9.0.3.

cd tests && DS_ACCELERATOR=cpu pytest unit/v1/half_precision/test_bf16.py -k StochasticRound

Result: 8 passed. The whole file, unit/v1/half_precision/test_bf16.py: 35 passed, 6 skipped.

Test Configurations What it checks
TestBF16StochasticRoundParams (2 ranks) BF16_Optimizer (ZeRO-1, fp32 grad accumulation); ZeRO-2; ZeRO-2 + CPU optimizer offload; each with the key off and on The test asserts the expected optimizer type and offload mode, so the configured path is the one under test. 65,537 elements (the last partition ends in padding) on the bf16 grid with |w| in [0.3, 1), constant gradient, 8 Adam steps at lr 1e-4: every master moves by 8e-4 (checked against the analytic value to 1e-6), less than half a spacing. Off: the bf16 parameters equal the masters rounded to nearest and have not moved. On: every element is one of its master's two bf16 neighbours (from torch.nextafter), the summed rounding error is within 5 standard errors of zero, and all ranks hold identical parameters.
TestBF16StochasticRoundParamsUnsupported (1 rank) ZeRO-3; ZeRO-2 with bf16_master_weights_and_grads deepspeed.initialize raises ValueError naming the key instead of silently ignoring it.

Red checks, each run on a copy of this branch with one part reverted:

Reverted part Result
The rounding (copy falls back to copy_) The 3 key-on cases fail: summed rounding error 17,221 against a standard error of 110 (156 standard errors). The 3 key-off cases still pass.
The engine refusal The ZeRO-3 case no longer raises: initialize returns, and the worker then did not exit before the 7- and 20-minute timeouts. It never passes, but it does not fail cleanly either.
The config refusal The bf16_master_weights_and_grads case fails.

pre-commit run --files <changed files>: all hooks pass.

Not validated:

  • Any GPU or other accelerator, including CUDA generators.
  • Convergence, throughput and peak memory (planned below).
  • Pipeline parallelism, MoE expert-parallel groups, immediate_grad_update, torch_autocast, Muon and universal-checkpoint load with the key on. Each reaches the same copy sites, by source reading only.
  • NVMe optimizer offload.

Planned experiments

The option changes convergence, so per CONTRIBUTING it needs full-training comparisons against round-to-nearest. Proposed, for maintainers to adjust:

  1. Fine-tuning a small LM at low learning rates (1e-6 to 1e-5) with ZeRO-2, round-to-nearest vs stochastic: loss curves and final evaluation, several seeds.
  2. An RL post-training run on a small model at lr 1e-6, close to the paper's setting.
  3. Throughput and peak memory on one GPU node, ZeRO-2 with and without optimizer offload, option off vs on.

Open question for maintainers: should the first version round on the accelerator under CPU offload (double the host-to-device bytes on that path) instead of paying the serial host draw?

AI assistance

This change, its tests and this description were written with AI assistance (Claude). The human submitter reviews every line, runs the device tests and signs off before this leaves draft.

…ters

ZeRO-1/2 and BF16_Optimizer copy each rank's fp32 master partition into
its bf16 parameters with round-to-nearest after every optimizer step. A
master update smaller than half a bf16 spacing does not change the bf16
parameter, so at small learning rates the parameters used by the forward
pass and saved in weights-only exports trail the masters by many steps.

Add bf16.stochastic_round_params (default false). When set, that copy
rounds each element to the bf16 neighbour away from zero with probability
equal to its fractional position between its two bf16 neighbours, so the
bf16 parameters equal the masters in expectation. The rounding happens
before the existing all-gather, so it adds no communication. Draws are
seeded from the torch seed and the data-parallel rank, so tensor-parallel
replicas of a parameter round identically. The engine refuses the key for
optimizers that do not apply it (ZeRO-3, ZenFlow, FP16_Optimizer), and
the config refuses it with bf16_master_weights_and_grads.

Test: two CPU ranks, eight Adam steps on a constant gradient that move
every master less than half a spacing. With the key off, the bf16
parameters stay at round-to-nearest of the masters (unchanged); with it
on, they are unbiased within 5 standard errors, for BF16_Optimizer and
ZeRO-2 with and without CPU offload.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@gpu-test-labeler gpu-test-labeler Bot added the skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. label Nov 20, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant