submission_status_cleanup only recovers Running submissions — Submitted, Preparing, and Scoring hang forever

[hanane-ca]

Bug

submission_status_cleanup() in src/apps/competitions/tasks.py only filters submissions with status=Running:

submissions = Submission.objects.filter(status=Submission.RUNNING, has_children=False)

Submissions stuck in Submitted, Preparing, or Scoring are never recovered by the periodic cleanup job. They hang indefinitely and permanently consume the participant's quota.

How it happens

A submission can get stuck in a non-Running state when:

• Submitted: the broker message is lost or the compute worker never picks it up
• Preparing: the worker crashes or is killed during setup (e.g. Docker pull failure)
• Scoring: the prediction→scoring re-enqueue fails (related to Worker status to FAILED instead of SCORING or FINISHED in case of failure #2030, Compute worker issues #1205, Submissions with exit code != 0 are marked as completed instead of failed #1641) — the status row commits as Scoring but the Celery task is never published

The existing cleanup only covers Running with a 24h + execution_time_limit threshold. All other non-terminal states are ignored.

Impact (observed in EEG Foundation Challenge, NeurIPS 2025)

• 8,328 total submissions, 51% never scored
• 3 submissions were still stuck in Preparing at the time of data export (months later)
• Stuck submissions consume quota (models.py:342 excludes only Failed, not Cancelled/Scoring/etc.), so participants lose attempts permanently
• The existing started_when check also crashes with a NoneType comparison for submissions that never reached Running (no started_when value)

Fix

Extend the filter to all non-terminal states (Submitted, Preparing, Running, Scoring) and use created_when as fallback when started_when is null.

Related

• PR Worker status to FAILED instead of SCORING or FINISHED in case of failure #2030 (improves worker-side failure handling but does not address cleanup coverage)
• Issues Compute worker issues #1205, Submissions with exit code != 0 are marked as completed instead of failed #1641