Skip to content

perf: reduce fixture dependency collection overhead - #15060

Open
host452b wants to merge 2 commits into
pytest-dev:mainfrom
host452b:perf/collection-fixture-lookup
Open

host452b wants to merge 2 commits into
pytest-dev:mainfrom
host452b:perf/collection-fixture-lookup

Conversation

@host452b

Copy link
Copy Markdown

Large unittest suites repeatedly traverse the same small fixture dependency graphs during collection. This reduces that per-item overhead without caching fixture state or deferring test-case construction.

  • Replace the recursive nested generator with an explicit DFS stack, preserving dependency order and fixture override indices while avoiding generator allocations and their reference cycle.
  • Skip string nodeid lookups and temporary lists when the relevant fixtures do not use legacy registration; retain the existing fallback for legacy fixtures.
  • Add collection regressions for dependency chains longer than the Python recursion limit, with and without direct parametrization, and a 35,000-method unittest benchmark. Runtime fixture setup still has separate recursion constraints.

Measurements

Five alternating baseline/patched pairs, Python 3.12.3, median elapsed time for the complete pytest --collect-only -q subprocess (including startup and shutdown), without profiling hooks:

Workload Baseline Patched Reduction
PyTorch test_ops.py, 34,525 items 14.806 s 14.306 s 3.4%
bench/unittest_methods.py, 35,000 items 2.734 s 2.273 s 16.8%

Ranges: PyTorch 14.501–15.133 s / 14.066–14.377 s; generated unittest methods 2.672–2.853 s / 2.213–2.372 s. Every run exited successfully with identical ordered nodeids within its workload. These are local measurements, not a general speedup guarantee.

Measurements compare commit 99ab2acccff15f757260a1e1fbb0d2a47fc01041 with this fixture optimization; the branch was subsequently updated to current main. PyTorch was 2.12.0a0+0291f960b6.nv26.04.48445190 with CPU-only generation. --noconftest was necessary because that checkout uses a hook argument removed in pytest 9. External plugin autoload and the cache provider were disabled. This does not measure the complete PyTorch CI/plugin environment or execute its test bodies; module imports and dynamic test generation remain major costs.

To reproduce the generated workload, use an otherwise empty pytest ini file and run the same command on baseline and patched checkouts:

PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 PYTHONDONTWRITEBYTECODE=1 PYTHONHASHSEED=0 \
  python -m pytest -c /path/to/empty-pytest.ini --noconftest \
  -p no:cacheprovider --collect-only -q bench/unittest_methods.py

Use the identical benchmark file in both checkouts and alternate the runs.

Validation

  • Full suite before the upstream update: 4,607 passed, 50 skipped, 13 xfailed, 7 xpassed; two existing deprecated-fixture warnings.
  • Both new regression cases fail on the unpatched baseline with RecursionError.
  • 10,000 generated fixture graphs produced identical ordered closures and fixture lookup sequences against the original implementation.
  • After updating to main 6a0de9be5: fixture, unittest, fixture-display and cache tests — 391 passed, 11 skipped, 3 xfailed; configured pre-commit checks, including ruff and mypy, passed.

AI assistance: implementation, tests, and benchmark investigation were prepared with OpenAI Codex, credited in the commit trailer.

host452b and others added 2 commits September 18, 2026 07:08
Use an explicit DFS stack for fixture closures and avoid legacy nodeid
lookups when fixtures use node-based registration. Preserve override
ordering and add deep-closure collection tests and a unittest benchmark.

Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
@host452b
host452b marked this pull request as ready for review September 18, 2026 07:08
@psf-chronographer psf-chronographer Bot added the bot:chronographer:provided (automation) changelog entry is part of PR label Sep 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bot:chronographer:provided (automation) changelog entry is part of PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant