Skip to content

fix: preserve incomplete flash checkpoints - #9963

Open
LeterTao wants to merge 1 commit into
modelscope:mainfrom
LeterTao:fix/flash-ckpt-trainer-state
Open

fix: preserve incomplete flash checkpoints#9963
LeterTao wants to merge 1 commit into
modelscope:mainfrom
LeterTao:fix/flash-ckpt-trainer-state

Conversation

@LeterTao

Copy link
Copy Markdown

Problem

When Flash Checkpoint is enabled, Swift writes checkpoint metadata such as
trainer_state.json before calling DLRover to persist the checkpoint.

If DLRover returns success=False, it may only indicate that asynchronous
persistence has not completed yet. However, Swift currently removes the entire
checkpoint directory in this case, deleting trainer_state.json and other
already-written files.

As a result, checkpoint recovery may fail even though the checkpoint is still
being persisted asynchronously.

Fix

Stop recursively deleting the checkpoint directory when DLRover reports that
the latest checkpoint is not finished.

The existing dlrover_latest.txt mechanism remains responsible for identifying
the latest complete checkpoint. Incomplete checkpoint directories are preserved
but are not selected during resume.

Validation

  • python -m compileall -q swift/trainers/mixin.py
  • git diff --check

@tastelikefeet

Copy link
Copy Markdown
Collaborator

Hi, thanks for your contribution! Can you check if this PR works in your scenario:https://github.com/modelscope/ms-swift/pull/9990/changes#diff-52addf56cb07f73a14e0e9c60c0f533c601ff29e09b7e22b8540fc93bb851d70R727

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants