Skip to content

[CI] Update test fetcher script - #14607

Open
DN6 wants to merge 4 commits into
mainfrom
update-tests-fetcher-script
Open

[CI] Update test fetcher script#14607
DN6 wants to merge 4 commits into
mainfrom
update-tests-fetcher-script

Conversation

@DN6

@DN6 DN6 commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator

What does this PR do?

Migrating PR CI to use the test fetcher by default. Doing this in parts to keep review manageable and testable.

First step is to just update the fetcher script.

The test fetcher determines which tests to run based on which files have been changed in a PR. It works in the following way

  1. First fetch the changed files based on the PR diff
  2. Read all the files under diffusers/src and tests/ to build a reverse dependency map. This map is a dictionary from a file in our repo to every file that would be affected if the file were changed. It is built in the following way
    1. For every .py under src/diffusers/ and tests/ return the files it imports. e.g. test_pipeline_flux.py -> [src/diffusers/pipelines/flux/pipeline_flux.py,...]
    2. Each file's list is expanded with its dependencies' dependencies, until this list stops growing. e.g. test_pipeline_flux.py's list also contains transformer_flux.py, attention.pyand so on, because these are modules are imported by pipeline_flux.py
    3. Invert this forward map. Each file entry's list becomes a key in the reverse map that maps back to the original file. So src/diffusers/pipelines/flux/pipeline_flux.py -> [tests/pipelines/flux/test_pipeline_flux.py, tests/pipelines/flux/test_pipeline_flux_control.py, ...]
  3. Look up the the changed files from the diff against reverse dependency map and filter just the test files to give us a final set of tests.
  4. This set of tests is then bucketed based on test type. e.g. pipelines, models, schedulers, etc.
  5. For models and pipelines. The fetcher parses the test file to learn which Mixin class carries which marker (LoraTesterMixin → lora, MemoryTesterMixin → memory, cpu_offload, group_offload, …). It uses this to construct a feature based matrix for pipeline/models tests so that the actual tests jobs run in feature based buckets. e.g model-core, model-memory, model-attention, pipeline-core, pipeline-lora. Here is an example of a workflow running this new test matrix: https://github.com/huggingface/diffusers/actions/runs/32945379597/job/98105816662

Once the fetcher script is merged into main, we can update the relevant workflows and test them side by side with the existing CI to ensure no gaps. FYI the fetcher tends to over select tests rather than miss them.

@github-actions github-actions Bot added tests CI size/L PR with diff > 200 LOC labels Aug 26, 2026
…sformer_flux.py` instead of a separate `utils.py`
@DN6
DN6 requested review from sayakpaul and yiyixuxu and a lite review from Copilot and removed request for Copilot August 26, 2026 10:43
release_memory(pipe)

def test_not_empty_state_dict(self):
from diffusers import AutoPipelineForText2Image

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fetcher decides "this test depends on file X" by looking at the imports at the top of the test file. AutoPipelineForText2Image lives in auto_pipeline.py, and that file imports every pipeline in the library. So if a test file imports AutoPipelineForText2Image at the top, the fetcher concludes that test depends on every pipeline, and any PR touching any pipeline would run it.

assert False, "Parameters not the same!"

def test_local_files_only_with_sharded_checkpoint(self):
from diffusers.models import FluxTransformer2DModel

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar issue as AutoPipelines. Importing Flux and SD3 at the top of the file suggests to the test fetcher that these model objects are a dependency for anything that imports from this file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI size/L PR with diff > 200 LOC tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant