[bug]: Race condition in test_download_queue.py #103
Description
Is there an existing issue for this problem?
- I have searched the existing issues
Install method
Invoke's Launcher
Operating system
Linux
GPU vendor
Nvidia (CUDA)
GPU model
No response
GPU VRAM
No response
Version number
main branch
Browser
No response
System Information
No response
What happened
I am seeing random failures of the unit test tests/app/services/download/test_download_queue.py crom what appears to be a race condition. Here is a typical stack trace:
=================================== FAILURES ===================================
_________________________________ test_errors __________________________________
tmp_path = PosixPath('/tmp/pytest-of-runner/pytest-0/test_errors0')
mm2_session = <requests_testadapter.TestSession object at 0x7f873fcc87d0>
@pytest.mark.timeout(timeout=10, method="thread")
def test_errors(tmp_path: Path, mm2_session: Session) -> None:
queue = DownloadQueueService(
requests_session=mm2_session,
)
queue.start()
for bad_url in ["http://www.civitai.com/models/broken", "http://www.civitai.com/models/missing"]:
queue.download(AnyHttpUrl(bad_url), dest=tmp_path)
queue.join()
jobs = queue.list_jobs()
print(jobs)
assert len(jobs) == 2
jobs_dict = {str(x.source): x for x in jobs}
assert jobs_dict["http://www.civitai.com/models/broken"].status == DownloadJobStatus.ERROR
> assert jobs_dict["http://www.civitai.com/models/broken"].error_type == "HTTPError(NOT FOUND)"
E assert "FileNotFound...downloading')" == 'HTTPError(NOT FOUND)'
E
E - HTTPError(NOT FOUND)
E + FileNotFoundError([Errno 2] No such file or directory: '/tmp/pytest-of-runner/pytest-0/test_errors0/missing.txt.downloading')
tests/app/services/download/test_download_queue.py:78: AssertionError
----------------------------- Captured stdout call -----------------------------
[DownloadJob(id=0, dest=PosixPath('/tmp/pytest-of-runner/pytest-0/test_errors0'), download_path=PosixPath('/tmp/pytest-of-runner/pytest-0/test_errors0/missing.txt'), status=<DownloadJobStatus.ERROR: 'error'>, bytes=0, total_bytes=0, error_type="FileNotFoundError([Errno 2] No such file or directory: '/tmp/pytest-of-runner/pytest-0/test_errors0/missing.txt.downloading')", error='Traceback (most recent call last):\n File "/home/runner/work/InvokeAI/InvokeAI/invokeai/app/services/download/download_default.py", line 316, in _download_next_item\n self._do_download(job)\n File "/home/runner/work/InvokeAI/InvokeAI/invokeai/app/services/download/download_default.py", line 391, in _do_download\n resume_from = candidates[0].stat().st_size\n ^^^^^^^^^^^^^^^^^^^^\n File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/pathlib.py", line 1013, in stat\n return os.stat(self, follow_symlinks=follow_symlinks)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nFileNotFoundError: [Errno 2] No such file or directory: \'/tmp/pytest-of-runner/pytest-0/test_errors0/missing.txt.downloading\'\n', source=AnyHttpUrl('http://www.civitai.com/models/broken'), access_token=None, priority=10, job_started='2026-02-28T16:13:25.053048+00:00', job_ended='2026-02-28T16:13:25.083713+00:00', content_type=None, canonical_url=None, etag=None, last_modified=None, final_url=None, expected_total_bytes=None, resume_required=False, resume_message=None, resume_from_scratch=False), DownloadJob(id=1, dest=PosixPath('/tmp/pytest-of-runner/pytest-0/test_errors0'), download_path=PosixPath('/tmp/pytest-of-runner/pytest-0/test_errors0/missing.txt'), status=<DownloadJobStatus.COMPLETED: 'completed'>, bytes=22, total_bytes=0, error_type=None, error=None, source=AnyHttpUrl('http://www.civitai.com/models/missing'), access_token=None, priority=10, job_started='2026-02-28T16:13:25.052943+00:00', job_ended='2026-02-28T16:13:25.054633+00:00', content_type=None, canonical_url=None, etag=None, last_modified=None, final_url='http://www.civitai.com/models/missing', expected_total_bytes=0, resume_required=False, resume_message=None, resume_from_scratch=False)]
----------------------------- Captured stderr call -----------------------------
[2026-02-28 16:13:25,083]::[DownloadQueueService]::ERROR --> http://www.civitai.com/models/broken: ['Traceback (most recent call last):\n', ' File "/home/runner/work/InvokeAI/InvokeAI/invokeai/app/services/download/download_default.py", line 316, in _download_next_item\n self._do_download(job)\n', ' File "/home/runner/work/InvokeAI/InvokeAI/invokeai/app/services/download/download_default.py", line 391, in _do_download\n resume_from = candidates[0].stat().st_size\n ^^^^^^^^^^^^^^^^^^^^\n', ' File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/pathlib.py", line 1013, in stat\n return os.stat(self, follow_symlinks=follow_symlinks)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', "FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pytest-of-runner/pytest-0/test_errors0/missing.txt.downloading'\n"]
------------------------------ Captured log call -------------------------------
ERROR DownloadQueueService:download_default.py:651 http://www.civitai.com/models/broken: ['Traceback (most recent call last):\n', ' File "/home/runner/work/InvokeAI/InvokeAI/invokeai/app/services/download/download_default.py", line 316, in _download_next_item\n self._do_download(job)\n', ' File "/home/runner/work/InvokeAI/InvokeAI/invokeai/app/services/download/download_default.py", line 391, in _do_download\n resume_from = candidates[0].stat().st_size\n ^^^^^^^^^^^^^^^^^^^^\n', ' File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/pathlib.py", line 1013, in stat\n return os.stat(self, follow_symlinks=follow_symlinks)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', "FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pytest-of-runner/pytest-0/test_errors0/missing.txt.downloading'\n"]
=============================== warnings summary ===============================
../../../../../opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/torch/amp/autocast_mode.py:266
/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/torch/amp/autocast_mode.py:266: UserWarning: User provided device_type of 'cuda', but CUDA is not available. Disabling
warnings.warn(
../../../../../opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/passlib/utils/__init__.py:854
/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/passlib/utils/__init__.py:854: DeprecationWarning: 'crypt' is deprecated and slated for removal in Python 3.13
from crypt import crypt as _crypt
tests/app/invocations/test_is_optional.py:7
/home/runner/work/InvokeAI/InvokeAI/tests/app/invocations/test_is_optional.py:7: PytestCollectionWarning: cannot collect test class 'TestModel' because it has a __init__ constructor (from: tests/app/invocations/test_is_optional.py)
class TestModel(BaseModel):
tests/app/routers/test_session_queue_sanitization.py:16
/home/runner/work/InvokeAI/InvokeAI/tests/app/routers/test_session_queue_sanitization.py:16: PytestCollectionWarning: cannot collect test class 'TestSanitizationInvocationOutput' because it has a __init__ constructor (from: tests/app/routers/test_session_queue_sanitization.py)
@invocation_output("test_sanitization_output")
tests/app/routers/test_session_queue_sanitization.py:21
/home/runner/work/InvokeAI/InvokeAI/tests/app/routers/test_session_queue_sanitization.py:21: PytestCollectionWarning: cannot collect test class 'TestSanitizationInvocation' because it has a __init__ constructor (from: tests/app/routers/test_session_queue_sanitization.py)
@invocation("test_sanitization", version="1.0.0")
../../../../../opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/requests_testadapter.py:96
/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/requests_testadapter.py:96: PytestCollectionWarning: cannot collect test class 'TestSession' because it has a __init__ constructor (from: tests/app/services/model_install/test_missing_models.py)
class TestSession(requests.Session):
../../../../../opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/requests_testadapter.py:96
/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/requests_testadapter.py:96: PytestCollectionWarning: cannot collect test class 'TestSession' because it has a __init__ constructor (from: tests/app/services/model_install/test_model_install.py)
class TestSession(requests.Session):
../../../../../opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/requests_testadapter.py:96
/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/requests_testadapter.py:96: PytestCollectionWarning: cannot collect test class 'TestSession' because it has a __init__ constructor (from: tests/app/services/model_load/test_load_api.py)
class TestSession(requests.Session):
tests/app/services/model_install/test_missing_models.py::TestBulkDelete::test_delete_installed_model
/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/tempfile.py:934: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp2ocmgp5z'>
_warnings.warn(warn_message, ResourceWarning)
tests/app/services/model_install/test_missing_models.py::TestBulkDelete::test_delete_installed_model
/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/tempfile.py:934: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpvmnr33ft'>
_warnings.warn(warn_message, ResourceWarning)
tests/app/services/model_install/test_missing_models.py::TestBulkDelete::test_delete_installed_model
/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/tempfile.py:934: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp7xda2bj0'>
_warnings.warn(warn_message, ResourceWarning)
tests/app/services/model_install/test_missing_models.py::TestBulkDelete::test_delete_installed_model
/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/tempfile.py:934: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpqvs6npio'>
_warnings.warn(warn_message, ResourceWarning)
tests/app/services/model_install/test_missing_models.py::TestBulkDelete::test_delete_installed_model
/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/tempfile.py:934: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp7ljj6dvc'>
_warnings.warn(warn_message, ResourceWarning)
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED tests/app/services/download/test_download_queue.py::test_errors - assert "FileNotFound...downloading')" == 'HTTPError(NOT FOUND)'
- HTTPError(NOT FOUND)
+ FileNotFoundError([Errno 2] No such file or directory: '/tmp/pytest-of-runner/pytest-0/test_errors0/missing.txt.downloading')
= 1 failed, 878 passed, 102 skipped, 4 deselected, 13 warnings in 243.96s (0:04:03) =
Error: Process completed with exit code 1.
What you expected to happen
The test should succeed consistently.
How to reproduce the problem
No response
Additional context
No response
Discord username
No response