Skip to content

test: preserve image baselines across Matplotlib text renderers - #9201

Draft
agourakis82 wants to merge 2 commits into
Project-MONAI:devfrom
agourakis82:fix/matplotlib-311-image-baselines
Draft

agourakis82 wants to merge 2 commits into
Project-MONAI:devfrom
agourakis82:fix/matplotlib-311-image-baselines

Conversation

@agourakis82

@agourakis82 agourakis82 commented Oct 20, 2026 •

Copy link
Copy Markdown

Related to #9069.

Problem

With Matplotlib 3.11.2, the image comparisons in ThreadContainer.test_plot and three matshow3d cases fail against the existing reference PNGs. This patch selects references by renderer generation while keeping the existing comparison tolerances and legacy references.

Changes

  • Add get_matplotlib_baseline() to select the existing references before Matplotlib 3.11 and a separate matplotlib_3_11 directory from 3.11.0.dev0 onward.
  • Use that selector in the four affected image comparisons. Keep the historical test_samples RMS expectation only for the legacy reference; the modern reference must satisfy the normal image comparison.
  • Add tests for the version boundary, prereleases and the absence of fallback when a modern reference is missing.
  • Exclude the new Matplotlib-dependent selector module from the minimal-dependency suite, following the contribution guidelines.
  • Include four proposed modern reference PNGs. Production MONAI modules, dependency bounds and CI configuration are unchanged by this PR.

Validation and scope

On 2026-10-24, a fresh Ubuntu 24.04 / CPython 3.12.3 environment with Torch 2.8.0, torchvision 0.23.0 and Matplotlib 3.11.2 reproduced the four affected image failures. The native MONAI extension was built and imported, and dependency checks passed.

The command selected the zarr merger, FID, threadcontainer and matshow3d test modules:

python -m unittest -v tests.inferers.test_zarr_avg_merger tests.metrics.test_compute_fid_metric tests.data.test_threadcontainer tests.visualize.utils.test_matshow3d
  • Before: 32 tests, four image-comparison failures.
  • After applying the exact published renderer patch and adding tests.test_matplotlib_baseline to the command: 34 tests passed, with no skips.
  • All eight renderer files, including the four PNGs, were checked against the published Git blobs at 9cbaeadfda56acc8e8a6cf973a59cf28e9a6ffd7. The follow-up changes only the minimal-suite exclusion list.

This execution used a private composition based on #9140 at 4043f3bd332bf4867fe5a3ac80abb5257841a2c9, with a separate CI experiment, followed by the exact renderer patch. It is focused regression evidence, not a hosted-CI or complete-suite result for this standalone PR. The follow-up minimal-suite exclusion was checked for a unique entry and an existing module; the rendering tests were not rerun for that registration change. Hosted workflow approval remains a separate gate.

Remaining draft item: reference distribution

The current draft commits the four PNGs directly. CONTRIBUTING.md asks contributors to host binary test data outside the source repository, preferably in MONAI-extra-test-data releases, and reference it through tests/testing_data/data_config.json.

The references need to be moved to that distribution path before this draft is ready. Could these four files be hosted in a MONAI-extra-test-data release so the tests can use the project's download helpers?

Types of changes

  • Test-only change.
  • New selector tests.
  • Full integration suite passed for this standalone PR.
  • Hosted CI passed.
  • Reference distribution aligned with the project guidelines.

Technical summary prepared with OpenAI Codex assistance.

Matplotlib 3.11 changes text rendering, so references must track
renderer generations without weakening image comparisons.

Related to Project-MONAI#9069.

Assisted-by: OpenAI Codex
Signed-off-by: Demetrios Agourakis <demetrios@agourakis.med.br>
@coderabbitai

coderabbitai Bot commented Oct 20, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Register the optional-dependency test module alongside test_matshow3d in
the minimal-suite exclusion list, as required by CONTRIBUTING.md.

Verified one unique exclusion entry, an existing target module, and
git diff --check. This does not change the rendering code or references.

Assisted-by: OpenAI Codex
Signed-off-by: Demetrios Agourakis <demetrios@agourakis.med.br>
@agourakis82

Copy link
Copy Markdown
Author

The remaining draft item is distributing the four Matplotlib 3.11 reference PNGs through external test-data storage, as requested by CONTRIBUTING.md. I rechecked the current MONAI-extra-test-data release assets and did not find these four references. I also confirmed that the frozen asset Git blobs still match this PR's current head, 435ccec.

Could a maintainer host these exact files in a MONAI-extra-test-data release, or indicate another approved location? Once the actual URLs exist, I can replace the source-tree PNGs with entries in data_config.json and use the existing test-data download helpers.

File SHA-256
matshow3d_patch_test.png 8cc2654ebc6d7edf3ea4bc9e3de427d15aec30e3815ed49885dade805b9b291a
matshow3d_rgb_test.png bd1862c88521284191dcaa24b10d01d54ee7dd6a32356b44cd363410ae81dc0b
matshow3d_test.png 420dc126c3237c97c19875ba203b3f0bcb0c987f50336f59ff9b4dacab38fac9
threadcontainer_plot_test.png 3e99f27e5761212068914845208959b33231e7b72e5847ed582ec252951a4a68

The files are currently accessible under tests/testing_data/matplotlib_3_11/ in this PR. They have not been regenerated, and this transfer does not change image-comparison tolerances. No rendering tests were rerun for this asset-custody check.

Prepared with OpenAI Codex assistance.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant