Skip to content

feat: add LTX-2.5 flexible multi-modality training - #2

Draft
NancyFyong wants to merge 1 commit into
ltx2_5from
ltx2_5_training
Draft

feat: add LTX-2.5 flexible multi-modality training#2
NancyFyong wants to merge 1 commit into
ltx2_5from
ltx2_5_training

Conversation

@NancyFyong

@NancyFyong NancyFyong commented Aug 30, 2026

Copy link
Copy Markdown
Owner

What

Adds LTX-2.5 multi-modality flexible training to DiffSynth-Studio, following the repository's existing LTX-2/2.3 training flow. Built on top of the LTX-2.5 split-component inference pipeline.

Supports the public flexible conditioning modes:

  • T2AV / I2AV (first-frame)
  • Video prefix/suffix extension
  • A2V / V2A (Foley)
  • Video inpainting / outpainting
  • T2A
  • Audio prefix/suffix extension, audio inpainting
  • V2V / A2A / AV2AV IC-LoRA

Each run can use the Dev or Distilled split BF16 DiT, as LoRA or full-DiT fine-tuning (full fine-tuning via an 8-process BF16 DeepSpeed ZeRO-3 accelerate config).

Implementation

  • diffsynth/pipelines/ltx25_training.py: training-only LTX25FlexibleTrainingPipeline and config. Per-modality is_generated, VAE encoding of target/reference media, intrinsic conditioning masks, reference latent positions, and modality-selective masked flow-matching loss.
  • LTXModel.forward: supports an absent video modality so audio-only (T2A / audio extension) training works, preserving existing dual-modality inference.
  • LTX25AudioVideoPipeline.from_pretrained: subclass-safe classmethod so the training pipeline can reuse split-component loading.
  • examples/ltx2/model_training/train_ltx25.py: generic launcher with cache (sft:data_process) and DiT-only (sft:train) split, LoRA and full-DiT options.
  • examples/ltx2/model_training/ltx25/*.yaml: 15 per-mode training-strategy configs.
  • examples/ltx2/model_training/{lora,full}/LTX-2.5-flexible-splited.sh + full/accelerate_config_ltx2_5_deepspeed_zero3.yaml: LoRA and full-DiT recipes.

Validation (no download / no real training on large data)

  • All new Python sources compile; launcher --help works.
  • All 15 mode configs parse and pass config validation.
  • Dev/Distilled split ModelConfig composition builds for cache and train tasks.
  • CPU dummy DiT validates video-only, audio-only, frozen-cross-modality, reference-token, masked-loss, and backward paths.
  • LoRA and full (ZeRO-3) T2AV runs completed on the bundled example dataset (121-frame video + audio) over 10 epochs, with TensorBoard loss logging and epoch/step checkpoints saved.
  • LTX-2.5 DiT 4,091-key metadata contract and inference behavior unchanged.

Use

LoRA:

bash examples/ltx2/model_training/lora/LTX-2.5-flexible-splited.sh t2av dev

Full DiT (DeepSpeed ZeRO-3):

bash examples/ltx2/model_training/full/LTX-2.5-flexible-splited.sh t2av dev

LTX25_MODEL_ROOT, LTX25_DATA_ROOT, LTX25_METADATA, LTX25_NUM_EPOCHS and LTX25_DATASET_REPEAT are configurable.

Note

This draft stacks on the LTX-2.5 inference PR (upstream modelscope#1602, fork ltx2_5). Once that lands on main, this PR's base can be switched to main to show only the training diff.

Implements split-component LTX-2.5 training aligned with the public
flexible conditioning modes, on top of the portable inference pipeline.

- new LTX25FlexibleTrainingPipeline: per-modality generated/frozen
  conditioning, VAE encoding, intrinsic and reference conditions,
  masked flow-matching loss, and a subclass-safe from_pretrained
- optional video modality in LTXModel.forward and audio-only forward
- generic train_ltx25.py launcher plus 15 mode YAML configs
- LoRA and full-DiT recipes with an 8-process BF16 DeepSpeed ZeRO-3 config
- training-only torchcodec audio loading for the cached-data path
@NancyFyong
NancyFyong changed the base branch from main to ltx2_5 August 30, 2026 02:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants