When a custom collector creates an Item whose name contains brackets, --last-failed can omit that item even though it failed in the previous run. If another previously failing item has been fixed, pytest reports success while the bracket-named item still fails in a full run.
Reproduced on main at 99ab2ac (9.2.0.dev335+g99ab2accc), Linux x86_64, Python 3.12.3, pluggy 1.6.0. Plugin autoload is disabled. The reproducer only needs pytest and the standard library. The documented YAML collector exhibits the same behavior; pytest 9.1.1 handles that YAML scenario correctly.
Save this as reproduce.py and run it using the Python environment containing the current development version of pytest:
import json
import os
from pathlib import Path
import subprocess
import sys
import tempfile
conftest = '''import json
import pytest
def pytest_collect_file(parent, file_path):
if file_path.name == "test_cases.json":
return Cases.from_parent(parent, path=file_path)
class Cases(pytest.File):
def collect(self):
for name, passed in json.loads(self.path.read_text()).items():
yield Case.from_parent(self, name=name, passed=passed)
class Case(pytest.Item):
def __init__(self, *, passed, **kwargs):
super().__init__(**kwargs)
self.passed = passed
def runtest(self):
assert self.passed
'''
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
(root / 'conftest.py').write_text(conftest)
(root / 'pytest.ini').write_text('[pytest]\n')
cases = root / 'test_cases.json'
env = dict(os.environ, PYTEST_DISABLE_PLUGIN_AUTOLOAD='1')
results = []
for label, data, args in [
('initial', {'a_bad[one]': False, 'b_fixed': False}, []),
('last-failed after fixing b_fixed', {'a_bad[one]': False, 'b_fixed': True}, ['--lf']),
('full run with same input', {'a_bad[one]': False, 'b_fixed': True}, []),
]:
cases.write_text(json.dumps(data))
run = subprocess.run([sys.executable, '-m', 'pytest', '-v', '--tb=no', *args], cwd=root, env=env)
print(f'{label}: exit {run.returncode}', flush=True)
results.append(run.returncode)
print(f'Exit sequence: {results}', flush=True)
Expected exit sequence: [1, 1, 1] (initial failures, rerun after fixing only b_fixed, full run).
Actual exit sequence: [1, 0, 1]. The second invocation collects only one item:
collecting ... collected 1 item
run-last-failure: rerun previous 1 failure
test_cases.json::b_fixed PASSED
1 passed
The subsequent full run reports a_bad[one] FAILED and b_fixed PASSED.
The failure appears to be in NodeId identity across the string cache boundary. The live custom item has names=("a_bad[one]",), params=None; parsing its cached string gives names=("a_bad",), params="one". These have identical public nodeid strings but compare and hash differently. LFPluginCollWrapper filters items using these structured IDs. If there is only the bracket-named failure, the no-matching-failure fallback runs everything; the second, matching failure is what exposes the omission and false success.
I plan to follow up with a focused fix and regression tests. This report and reproducer were prepared with OpenAI Codex assistance; the exit sequence above was verified by running the script.
When a custom collector creates an Item whose name contains brackets,
--last-failedcan omit that item even though it failed in the previous run. If another previously failing item has been fixed, pytest reports success while the bracket-named item still fails in a full run.Reproduced on main at 99ab2ac (
9.2.0.dev335+g99ab2accc), Linux x86_64, Python 3.12.3, pluggy 1.6.0. Plugin autoload is disabled. The reproducer only needs pytest and the standard library. The documented YAML collector exhibits the same behavior; pytest 9.1.1 handles that YAML scenario correctly.Save this as
reproduce.pyand run it using the Python environment containing the current development version of pytest:Expected exit sequence:
[1, 1, 1](initial failures, rerun after fixing onlyb_fixed, full run).Actual exit sequence:
[1, 0, 1]. The second invocation collects only one item:The subsequent full run reports
a_bad[one] FAILEDandb_fixed PASSED.The failure appears to be in NodeId identity across the string cache boundary. The live custom item has
names=("a_bad[one]",), params=None; parsing its cached string givesnames=("a_bad",), params="one". These have identical public nodeid strings but compare and hash differently.LFPluginCollWrapperfilters items using these structured IDs. If there is only the bracket-named failure, the no-matching-failure fallback runs everything; the second, matching failure is what exposes the omission and false success.I plan to follow up with a focused fix and regression tests. This report and reproducer were prepared with OpenAI Codex assistance; the exit sequence above was verified by running the script.