Skip to content

TextToImage generate() path has no executable end-to-end test coverage (all asserts skipped) #1212

@msluszniak

Description

@msluszniak

Description

In common/rnexecutorch/tests/integration/TextToImageTest.cpp, every test that actually asserts on generate() output is skipped:

  • TextToImageGenerateTests.GenerateReturnsFileUri (line ~117) — GTEST_SKIP() "due to UNet forward call throwing error no. 1"
  • TextToImageGenerateTests.SameSeedProducesSameResult (line ~130) — same skip
  • The typed-suite TODO at line 54 notes TextToImage fails on MultipleGeneratesWork in the emulator environment

The skips are attributed to the emulator (UNet forward error no. 1). The net effect is that the entire TextToImage generation path — output file:// URI, seed determinism, and repeated generation — is currently unverified by any running test. The only assertions that execute are the negative/throw cases.

Two things to resolve:

  1. Coverage: establish an environment that runs these tests end-to-end (real device, or a CI/host backend that doesn't hit the emulator UNet limitation) so generation, determinism, and multiple-generate are actually exercised.
  2. Root cause: confirm whether "UNet forward error no. 1" in the emulator is purely environmental, or whether it masks a real defect in the UNet forward / scheduler integration. The MultipleGeneratesWork failure in particular is worth checking for state not being reset between successive generate() calls.

Steps to reproduce

  1. Build/run the native integration test suite under common/rnexecutorch/tests/integration in an emulator environment.
  2. Remove the GTEST_SKIP() lines in TextToImageGenerateTests.
  3. Observe the UNet forward call throwing error no. 1; note whether the typed-suite MultipleGeneratesWork case fails for the same reason or a distinct one.

Snack or a link to a repository

N/A — in-repo native test suite.

React Native Executorch version

main @ v0.8.4

React Native version

N/A — native (C++) integration test.

Platforms

Platform agnostic (shared C++); failure currently observed in emulator environments — confirm real-device behavior.

JavaScript runtime / Workflow / Architecture / Build type / Device

Device: emulator (where the skip applies). Otherwise N/A — native gtest.

AI model

TextToImage diffusion pipeline (tokenizer + text encoder + UNet + VAE decoder, per the test's constructor args).

Acknowledgements

Yes — searched existing issues; #939 (export/quantize a better T2I model) and #942 (img2img) are about models/features, not this coverage/skip gap.

Metadata

Metadata

Assignees

No one assigned

    Labels

    modelIssues related to exporting, improving, fixing ML modelstestIssue and PR related to tests or testing infrastructure

    Type

    No fields configured for Bug.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions