Repository navigation
Description
Bug description
DistributedCAME does not preserve its moment buffers between optimizer steps:
- For factored parameters, every non-initial step assigns exp_avg_res_row/col to exp_avg_sq_row/col. The confidence-residual estimates are discarded, and both statistics then update the same tensors.
- For unfactored parameters, update aliases exp_avg and is multiplied in place by the learning rate. This scales the stored first moment before the next step.
Both paths diverge from the ordinary CAME implementation starting with the second parameter update. The issue concerns the same optimizer-state preservation contract; the factored and unfactored paths should retain their own moment values.
Native reproduction
import torch
from colossalai.nn.optimizer import CAME, DistributedCAME
parameter = torch.nn.Parameter(torch.arange(1, 7, dtype=torch.float32).reshape(2, 3) / 10)
distributed_parameter = torch.nn.Parameter(parameter.detach().clone())
reference = CAME([parameter], lr=0.01)
optimizer = DistributedCAME([distributed_parameter], lr=0.01)
optimizer.setup_distributed(use_zero=False)
for step in range(3):
gradient = torch.tensor([[0.1, 0.2, 0.3], [0.4, 0.5, 0.6]]) * (step + 1)
parameter.grad = gradient.clone()
distributed_parameter.grad = gradient.clone()
reference.step()
optimizer.step()
state = optimizer.state[distributed_parameter]
print(step + 1, (parameter - distributed_parameter).abs().max().item(),
state["exp_avg_res_row"] is state["exp_avg_sq_row"])The maximum parameter difference is 0 on step 1, approximately 0.10983 on step 2 and 0.27012 on step 3. The residual/gradient buffers become the same objects on step 2.
For a vector with values/gradients [0.1, 0.2, 0.3], the first stored moment is about 0.001 instead of the reference 0.1 when lr=0.01. Parameters agree after the first update but diverge on subsequent updates.
Expected behavior
Keep the first, squared-gradient and confidence-residual moments intact between steps. Multi-step updates and checkpoint continuation should match CAME when configured without sharding.
Environment and history
- Main 297c8cf, Python 3.12, PyTorch 2.9.1+cpu.
- Real CAME/DistributedCAME objects, setup_distributed(use_zero=False). This is a native local reproduction, not a multi-rank test.
- Complete distributed_came.py history traces the residual alias to [Feature] Distributed optimizers: Lamb, Galore, CAME and Adafactor #5694; subsequent [Feature] auto-cast optimizers to distributed version #5746 work concerns optimizer casting.
- All-state exp_avg_res_row PR search found no result. The CAME/state results are [shardformer] Add explicit attention backends and packed varlen support #6508 (no optimizer files) and original proposal [optim] add distributed came #5526, whose vector path also scales the aliased first moment.
- No equivalent correction found in the user's pre-run contribution inventory or the inspected upstream history.
Activity