Skip to content

fix: move conditioning tensors to model device when partial loading enabled#8970

Open
aayushbaluni wants to merge 1 commit intoinvoke-ai:mainfrom
aayushbaluni:fix/8850-partial-loading-device-mismatch
Open

fix: move conditioning tensors to model device when partial loading enabled#8970
aayushbaluni wants to merge 1 commit intoinvoke-ai:mainfrom
aayushbaluni:fix/8850-partial-loading-device-mismatch

Conversation

@aayushbaluni
Copy link

Summary

Fixes #8850

When enable_partial_loading is true, conditioning embeddings from compel may remain on CPU while the UNet model is on CUDA. This causes a RuntimeError: Expected all tensors to be on the same device during denoising.

Root Cause

The _apply_standard_conditioning and _apply_standard_conditioning_sequentially methods pass conditioning tensors (.embeds, .pooled_embeds, .add_time_ids) directly to model_forward_callback without ensuring they are on the same device as the model input x. When partial loading offloads conditioning data to CPU to save VRAM, these tensors stay on CPU while the model expects CUDA tensors.

Fix

Before each model_forward_callback call, explicitly move conditioning tensors to x.device:

  • uncond_text.embeds and cond_text.embeds via .to(x.device)
  • added_cond_kwargs (SDXL text_embeds and time_ids) via .to(x.device)

Applied to both the batch path (_apply_standard_conditioning) and the sequential path (_apply_standard_conditioning_sequentially).

Test Plan

  1. Enable enable_partial_loading in InvokeAI config
  2. Generate an image using any SD/SDXL model
  3. Previously: crashes with RuntimeError: Expected all tensors to be on the same device
  4. Now: generation completes successfully

Made with Cursor

…nabled

When enable_partial_loading is true, conditioning embeddings from compel
may remain on CPU while the UNet model is on CUDA. Passing CPU tensors to
the CUDA model causes RuntimeError: Expected all tensors on same device.

Ensure all conditioning tensors (embeds, pooled_embeds, add_time_ids) are
moved to the same device as the input tensor x before calling
model_forward_callback, in both batch and sequential conditioning paths.

Fixes invoke-ai#8850
@github-actions github-actions bot added python PRs that change python files backend PRs that change backend files labels Mar 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend PRs that change backend files python PRs that change python files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[bug]: Runtime error when enable_partial_loading is enabled

1 participant