Skip to content

Return 503 when the database drops a request's connection #905

Open

Description

@bencap

Summary

When Postgres ends a session, for example after the API's 60-second idle-in-transaction limit or during a restart or failover, the next query fails and the API returns a 500 and sends a Slack alert. The client can't tell that retrying will work. This issue returns 503 with Retry-After for a dropped connection instead, and keeps the Slack alert.

Steps to reproduce

  1. Open a session with SET idle_in_transaction_session_timeout = '500ms'.
  2. Run a query, wait 1.5 seconds, run another.

Observed

sqlalchemy.exc.OperationalError: server closed the connection unexpectedly, with no SQLSTATE (err.orig.pgcode is None) and err.connection_invalidated set. CatchAllErrorMiddleware returns 500.

Fix

  • In lib/middleware/errors.py, treat a DBAPIError with connection_invalidated as a 503 with Retry-After: 5.
  • Unlike the database timeouts, keep sending the Slack alert: a dropped connection can also mean an outage.

Acceptance criteria

  • A route whose connection is closed by the server returns 503 with Retry-After.
  • The Slack alert still fires for it.
  • Other database errors still return 500.
Background

Postgres reports this timeout as SQLSTATE 25P03, but it closes the connection as it does, so the client only sees the dropped connection. Checked with psycopg2 against a local Postgres.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    app: backendTask implementation touches the backend

    Type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions