Skip to content

[BUG]: DistributedCAME corrupts optimizer moment buffers after updates #6546

Open

Description

@Afloat16

Bug description

DistributedCAME does not preserve its moment buffers between optimizer steps:

  • For factored parameters, every non-initial step assigns exp_avg_res_row/col to exp_avg_sq_row/col. The confidence-residual estimates are discarded, and both statistics then update the same tensors.
  • For unfactored parameters, update aliases exp_avg and is multiplied in place by the learning rate. This scales the stored first moment before the next step.

Both paths diverge from the ordinary CAME implementation starting with the second parameter update. The issue concerns the same optimizer-state preservation contract; the factored and unfactored paths should retain their own moment values.

Native reproduction

import torch
from colossalai.nn.optimizer import CAME, DistributedCAME

parameter = torch.nn.Parameter(torch.arange(1, 7, dtype=torch.float32).reshape(2, 3) / 10)
distributed_parameter = torch.nn.Parameter(parameter.detach().clone())
reference = CAME([parameter], lr=0.01)
optimizer = DistributedCAME([distributed_parameter], lr=0.01)
optimizer.setup_distributed(use_zero=False)

for step in range(3):
    gradient = torch.tensor([[0.1, 0.2, 0.3], [0.4, 0.5, 0.6]]) * (step + 1)
    parameter.grad = gradient.clone()
    distributed_parameter.grad = gradient.clone()
    reference.step()
    optimizer.step()
    state = optimizer.state[distributed_parameter]
    print(step + 1, (parameter - distributed_parameter).abs().max().item(),
          state["exp_avg_res_row"] is state["exp_avg_sq_row"])

The maximum parameter difference is 0 on step 1, approximately 0.10983 on step 2 and 0.27012 on step 3. The residual/gradient buffers become the same objects on step 2.

For a vector with values/gradients [0.1, 0.2, 0.3], the first stored moment is about 0.001 instead of the reference 0.1 when lr=0.01. Parameters agree after the first update but diverge on subsequent updates.

Expected behavior

Keep the first, squared-gradient and confidence-residual moments intact between steps. Multi-step updates and checkpoint continuation should match CAME when configured without sharding.

Environment and history

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions