Skip to content

Custom location returns 502 after backend Docker container is recreated (stale upstream not re-resolved) #5603

@Feeh03114

Description

@Feeh03114

Checklist

Summary

A custom location whose forward host points to a Docker container (by service/container name) returns 502 Bad Gateway after that backend container is recreated (e.g. a redeploy / docker compose up -d --force-recreate / image update), even though the main proxy host location keeps working. The custom location keeps the old (now invalid) upstream IP and never re-resolves the container name until the proxy host is manually re-saved (which regenerates and reloads the nginx config).

This is the same root cause as #302 (stale upstream after container recreation), but it specifically affects custom locations and is easy to miss because the primary location often recovers on its own while the custom location stays broken.

Steps to reproduce

  1. Create a Proxy Host with a main forward to a Docker container, e.g. web on the same Docker network.
  2. Add a Custom Location (e.g. /api/unicore) forwarding to a different backend container, e.g. api:8080.
  3. Confirm both https://host/ and https://host/api/unicore/... return 200.
  4. Recreate only the backend (api) container (redeploy / new image / --force-recreate). The container comes back healthy with a (possibly) new IP on the Docker network.
  5. Request the custom location again: https://host/api/unicore/...502 Bad Gateway (server: openresty), while https://host/ still returns 200.

Expected behavior

After the backend container is recreated, the custom location should re-resolve the container name (Docker embedded DNS) and route to the new IP, the same way it would after a manual proxy-host re-save.

Actual behavior

The custom location keeps the stale upstream IP and returns 502 until the proxy host is edited and saved again (which forces nginx to regenerate config and re-resolve the upstream). A container restart of NPM itself, or re-saving the proxy host, fixes it.

Workaround

Re-save the affected proxy host (or reload/restart the NPM container) after every backend redeploy so nginx re-resolves the Docker upstream. In our case we now do an NPM proxy-host refresh as a mandatory post-deploy step.

Root cause / suggestion

nginx resolves proxy_pass upstream names at config load time and caches the IP for the life of the worker. When NPM emits a static proxy_pass http://container_name:port; for a custom location, a recreated container with a new IP is never picked up.

Possible fixes:

  • Use the Docker embedded DNS resolver with a variable in proxy_pass so nginx re-resolves at request time, e.g.:
    resolver 127.0.0.11 valid=10s;
    set $upstream http://container_name:port;
    proxy_pass $upstream;
    (applied to custom locations as well as the main location)
  • This overlaps with the existing feature request to use Docker DNS / a dynamic resolver for upstreams.

Environment

  • NPM: latest
  • Deployment: Docker / docker compose, all containers on the same user-defined bridge network
  • Reverse proxy reports server: openresty

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions