feat: add LTX-2.5 flexible multi-modality training - #2
Draft
NancyFyong wants to merge 1 commit into
Draft
Conversation
Implements split-component LTX-2.5 training aligned with the public flexible conditioning modes, on top of the portable inference pipeline. - new LTX25FlexibleTrainingPipeline: per-modality generated/frozen conditioning, VAE encoding, intrinsic and reference conditions, masked flow-matching loss, and a subclass-safe from_pretrained - optional video modality in LTXModel.forward and audio-only forward - generic train_ltx25.py launcher plus 15 mode YAML configs - LoRA and full-DiT recipes with an 8-process BF16 DeepSpeed ZeRO-3 config - training-only torchcodec audio loading for the cached-data path
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds LTX-2.5 multi-modality flexible training to DiffSynth-Studio, following the repository's existing LTX-2/2.3 training flow. Built on top of the LTX-2.5 split-component inference pipeline.
Supports the public flexible conditioning modes:
Each run can use the Dev or Distilled split BF16 DiT, as LoRA or full-DiT fine-tuning (full fine-tuning via an 8-process BF16 DeepSpeed ZeRO-3
accelerateconfig).Implementation
diffsynth/pipelines/ltx25_training.py: training-onlyLTX25FlexibleTrainingPipelineand config. Per-modalityis_generated, VAE encoding of target/reference media, intrinsic conditioning masks, reference latent positions, and modality-selective masked flow-matching loss.LTXModel.forward: supports an absent video modality so audio-only (T2A / audio extension) training works, preserving existing dual-modality inference.LTX25AudioVideoPipeline.from_pretrained: subclass-safeclassmethodso the training pipeline can reuse split-component loading.examples/ltx2/model_training/train_ltx25.py: generic launcher with cache (sft:data_process) and DiT-only (sft:train) split, LoRA and full-DiT options.examples/ltx2/model_training/ltx25/*.yaml: 15 per-mode training-strategy configs.examples/ltx2/model_training/{lora,full}/LTX-2.5-flexible-splited.sh+full/accelerate_config_ltx2_5_deepspeed_zero3.yaml: LoRA and full-DiT recipes.Validation (no download / no real training on large data)
--helpworks.Use
LoRA:
Full DiT (DeepSpeed ZeRO-3):
LTX25_MODEL_ROOT,LTX25_DATA_ROOT,LTX25_METADATA,LTX25_NUM_EPOCHSandLTX25_DATASET_REPEATare configurable.Note
This draft stacks on the LTX-2.5 inference PR (upstream modelscope#1602, fork
ltx2_5). Once that lands onmain, this PR's base can be switched tomainto show only the training diff.