Skip to content

Fix batching and precomputed embeddings in Kandinsky 5 pipelines#14168

Draft
pzarzycki wants to merge 1 commit into
huggingface:mainfrom
pzarzycki:fix/kandinsky5-batch-embeddings
Draft

Fix batching and precomputed embeddings in Kandinsky 5 pipelines#14168
pzarzycki wants to merge 1 commit into
huggingface:mainfrom
pzarzycki:fix/kandinsky5-batch-embeddings

Conversation

@pzarzycki

Copy link
Copy Markdown
Contributor

What does this PR do?

Kandinsky 5 pipelines expanded the latent batch for num_images_per_prompt and num_videos_per_prompt, but did not always expand the matching text and visual conditioning. This caused batch mismatches when generating multiple outputs, especially when callers supplied precomputed prompt embeddings.

This PR makes batching consistent across the T2I, I2I, T2V, and I2V pipelines. It:

  • repeats Qwen and CLIP embeddings in per-prompt output order;
  • rebuilds cumulative Qwen sequence lengths for the expanded batch;
  • supports precomputed positive and negative embeddings with multiple outputs per prompt;
  • handles classifier-free guidance when prompt=None;
  • passes num_videos_per_prompt through the T2V text-encoding path;
  • expands I2I and I2V visual conditioning to the effective batch size;
  • accepts PIL, NumPy, tensor, and list image inputs in the I2I prompt-encoding path;
  • validates that precomputed embedding components have consistent batch sizes.

Focused regression tests cover all four pipelines. This addresses the batching and precomputed-embedding findings in #13639.

Tests

  • Full Kandinsky 5 suite excluding the existing unsupported fp16 cases: 122 passed, 16 skipped, 4 deselected
  • Focused new and changed CPU regressions: 9 passed
  • CUDA fp32 smoke matrix for T2I, I2I, T2V, and I2V with precomputed CFG and two outputs per prompt
  • CUDA bf16-autocast smoke matrix for all four pipelines
  • make fix-copies
  • Ruff check and formatting
  • python -m compileall -q src/diffusers/pipelines/kandinsky5 tests/pipelines/kandinsky5
  • git diff --check

Before submitting

  • Did you use an AI agent (Claude Code, Codex, Cursor, etc.) to help with this PR? If so:
  • Did you read the contributor guideline?
  • Did you read our philosophy doc? (important for complex PRs)
  • Was this discussed/approved via a GitHub issue or the forum? See kandinsky5 model/pipeline review #13639.
  • Did you make sure to update the documentation with your changes? No public arguments changed; existing docstrings already describe the supported inputs.
  • Did you write any new necessary tests?
  • Are you the author (or part of the team) of the model/pipeline (only applicable for model/pipeline related PRs)?

Who can review?

@yiyixuxu @asomoza

@github-actions github-actions Bot added tests pipelines size/L PR with diff > 200 LOC labels Jul 11, 2026
@pzarzycki

Copy link
Copy Markdown
Contributor Author

While running the full Kandinsky 5 test suite on CUDA, I noticed that the generic test_float16_inference fails for the T2V and I2V pipelines.

I reproduced both failures on a clean checkout of the same main commit, so they are unrelated to this PR. The generic test casts the entire pipeline with .to(torch.float16), while Kandinsky5Transformer3DModel declares several modules that should remain in fp32. The T2I and I2I test classes already skip this test, but T2V and I2V do not.

I have two ideas:

  1. Add matching fp16 skips for T2V and I2V in this PR.
  2. Handle the test inconsistency separately and keep this PR focused on batching.

What do you think?

CUDA fp32 and bf16-autocast inference pass for all four Kandinsky 5 pipelines.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pipelines size/L PR with diff > 200 LOC tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant