Skip to content

test(transforms): cover all four shadow-mask directions deterministically - #2144

Open
linhongyu510 wants to merge 1 commit into
mindee:mainfrom
linhongyu510:fix/deterministic-shadow-mask-coverage
Open

linhongyu510 wants to merge 1 commit into
mindee:mainfrom
linhongyu510:fix/deterministic-shadow-mask-coverage

Conversation

@linhongyu510

Copy link
Copy Markdown

What's wrong

create_shadow_mask picks one of four intensity gradients from an unseeded draw, and branches on it:

_params = np.random.rand(1)
quad_idx = int(_params[0] / 0.25)   # 0, 1, 2 or 3
if quad_idx % 2 == 0:              # line 173
    intensity_mask = ...
    if quad_idx == 0:              # line 175
        intensity_mask = 1 - intensity_mask
else:                              # line 178
    intensity_mask = ...
    if quad_idx == 1:              # line 180
        intensity_mask = 1 - intensity_mask

test_random_shadow calls it four times (its parametrization) with no seed, so each CI run reaches only some of those lines. Over 400 seeds the four quadrants come up 104 / 99 / 95 / 102 times, so which lines get counted is close to a coin flip.

The visible effect is that coverage of doctr/transforms/functional/base.py moves on its own. Measured with coverage.py, running the existing test 12 times and changing nothing at all:

missing lines within 165-185 : {(173, 174, 175): 1, (180,): 3, (): 5, (175,): 3}
total missing lines          : {15: 1, 16: 7, 19: 1, 28: 1, 29: 2}
deterministic?               : NO

Four different miss patterns, and the file's miss count swinging between 15 and 29.

Why it matters beyond this file

That swing lands on unrelated pull requests as a spurious codecov/project failure. Two examples from this repository's own history:

commit on main what it changed misses coverage
60cb879 Correct some vocabs 276 97.25%
1a6f4c1 Update docs — no code at all 278 97.23%
c7f7218 Correct vocabs (#2139) 278 97.23%

A docs-only commit moved project coverage by 2 misses / 0.02%.

The same thing hit #2138, where Codecov's per-file data attributes the whole project delta to this file even though the PR never touches it (has_diff: false, patch: null, line count unchanged 68 → 68, misses 4 → 6) while all three files that PR does touch keep their miss counts exactly.

Fix

Pin the two draws with monkeypatch so all four gradients are exercised on every run. After the change, 12 repeated runs:

missing lines within 165-185 : {(): 12}
total missing lines          : {15: 12}
deterministic?               : YES

One pattern, one count, every time — and line 180 (quad_idx == 1) is now covered where the existing test left it to chance.

I chose monkeypatch over np.random.seed(...) deliberately: a fixed seed only reaches whichever quadrant that seed happens to produce, and the mapping from seed to quadrant is an implementation detail of numpy's RNG stream that can change between versions. Patching the draw states the intent directly — "exercise the top-to-bottom gradient" — and keeps working if numpy's generator changes.

The existing test_random_shadow is left exactly as it is; this is an addition, not a replacement, so the end-to-end random path stays covered.

Verification

pytest tests/pytorch/test_transforms_pt.py -k shadow      8 passed  (4 existing + 4 new)
pytest tests/pytorch/test_transforms_pt.py               35 passed
ruff check / ruff format --check                          clean

Diff is +37/-0 in one test file; no source change.

Each of the four cases asserts the mask shape, that values stay in [0, 1], that the mask is a real gradient (min < max, not a constant plane), and that exactly two draws were consumed — so the test fails loudly if create_shadow_mask ever changes how it consumes randomness, rather than silently passing on a stale assumption.

…ally

create_shadow_mask picks one of four intensity gradients from an unseeded
np.random.rand(1), so test_random_shadow exercises only one of them per run
and the set of covered lines in transforms/functional/base.py changes between
runs. Measured over 12 runs, the miss count for that file swings between 15
and 29, which shows up as spurious codecov/project deltas on unrelated PRs.

Pin the draw with monkeypatch so every gradient is exercised on each run: the
miss set becomes identical across all 12 runs, and line 180 (the quad_idx == 1
branch) is now covered.
@codecov

codecov Bot commented Sep 20, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.25%. Comparing base (c7f7218) to head (d070254).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2144      +/-   ##
==========================================
+ Coverage   97.23%   97.25%   +0.01%     
==========================================
  Files         169      169              
  Lines       10039    10039              
==========================================
+ Hits         9761     9763       +2     
+ Misses        278      276       -2     
Flag Coverage Δ
unittests 97.25% <ø> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@linhongyu510

Copy link
Copy Markdown
Author

The docker check here is red, and it is not this PR — filed as #2145 with the full trace.

Short version: api/Dockerfile:18 runs make lock at build time and no lock file is committed, so each build re-resolves against current PyPI. nvidia-cublas==13.8.0.4 landed on 2026-09-16, and torch 2.14.0 now asks for it directly while also pulling cuda-toolkit==13.0.3, which pins nvidia-cublas==13.1.1.3.*. Both markers evaluate true on the image's platform, so pip gets two pins for one package and stops.

Checks that this is unrelated to the diff:

  • this branch changes one file, tests/pytorch/test_transforms_pt.py; nothing under api/
  • api/pyproject.toml was last modified on 2026-08-21
  • the last green docker run was 05d0615 on 09-18; mine on 09-19 is the first run after the upstream release
  • reproduced locally from just api/pyproject.toml + api/Makefile, with no doctr checkout involved

The other 8 workflows on this PR are green, including tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant