Skip to content

_update_submission() has no retry and _update_status() silently swallows exceptions, causing permanent submission loss #2415

@dconstancy

Description

@dconstancy

Bug

_update_submission() in compute_worker/compute_worker.py performs a single HTTP PATCH to report submission status back to Django. On any non-200 response, it raises SubmissionException immediately with no retry:

resp = self.requests_session.patch(url, data=data, timeout=150)
if resp.status_code == 200:
    logger.info("Submission updated successfully!")
else:
    raise SubmissionException("Failure updating submission data.")

The caller _update_status() then catches all exceptions and silently discards them:

try:
    self._update_submission(data)
except Exception as e:
    # Always catch exception and never raise error
    logger.exception(f"Failed to update submission status to {status}: {e}")

This means a single transient network error (502, timeout, connection reset) permanently loses the submission result. The compute worker moves on to the next job as if nothing happened, while Django never receives the status=Finished PATCH. The submission stays stuck in Scoring or Running forever.

How it happens

  1. Compute worker finishes scoring and sends PATCH /api/submissions/{id}/ {status: Finished}
  2. The network returns a transient error (502, timeout, connection reset)
  3. _update_submission() raises SubmissionException("Failure updating submission data.")
  4. _update_status() catches the exception, logs it, and does not re-raise
  5. The compute worker proceeds to the next submission
  6. Django never receives the Finished status → the submission stays stuck in Scoring indefinitely
  7. The participant sees a perpetually "processing" submission with no error message and no way to know it failed

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions