Skip to content

Refuse PUT provide requests at route-origin - #27

Merged
TakGN merged 1 commit into
mainfrom
origin-refuse-put-providers
Nov 2, 2026
Merged

TakGN merged 1 commit into
mainfrom
origin-refuse-put-providers

Conversation

@TakGN

@TakGN TakGN commented Nov 2, 2026

Copy link
Copy Markdown
Collaborator

Related to ipni/infra#247, which changes the 5xx alert thresholds. This can merge and deploy independently of it.

Why

needle can never succeed at PUT /routing/v1/providers/: ProvideBitswap always returns routing.ErrNotSupported, and boxo's server answers that with a 500. Envoy's /routing/v1/ route passes these requests through, so they count as server errors in both RouteOrigin5xx and NeedleServerErrors.

Measured on 2026-11-02, over one hour:

Box Share of needle's 500s on this route 5xx ratio Requests/s
sing-1 93% 4.2% 1.73
lith-1 92% 0.25% 11.79
chic-1 84% 0.15% 29.16

Every request on that route in that hour was a PUT answered with 500. sing-1 has little traffic, so a small stream of these was enough to warn: needle-c on sing1 answers 6.742% of requests with 5xx. Lookup probes were passing at the time.

Without this route, 5xx across the other needle routes was about 0.3% on sing-1 and 0.03% or less on the other two.

Change

k8s/route-origin/envoy.yaml.j2: a route above the others answers PUT on the /routing/v1/providers prefix with a 405, without reaching needle.

  • It has to be a 4xx. A 501 would still count as 5xx in RouteOrigin5xx.
  • The new provide-refused decorator is a new span name, in line with the file's other routes.
  • These requests no longer draw from the lookups rate limit bucket.
  • Clients sending these already got an error; they now get a 405 instead of a 500.

Checked

scripts/render-manifests.yml renders the template. The rendered config passes --mode validate on the pinned Envoy image (distroless-v1.39.1), and I ran that image locally with a self-signed certificate and no needle behind it:

Request Status
PUT /routing/v1/providers/, also without the slash and with a CID 405
GET /routing/v1/providers/<cid>, GET /routing/v1/peers/x 503, passed to the (absent) upstream
PUT /routing/v1/ipns/x, GET /routing/v1/ipns/x, GET /version 503, passed to the (absent) upstream
GET /debug/metrics/prometheus 404, still not exposed

Not checked: real traffic through Cloudflare.

After merging

This doesn't deploy by itself. routing.yml isn't applied from CI, so someone needs to run it. The ConfigMap hash rolls the Envoy pod on each box:

ansible-playbook routing.yml --tags origin --check --diff
ansible-playbook routing.yml --tags origin

Then RouteOrigin5xx and NeedleServerErrors should drop on every box, with sing-1 well under its thresholds.

🤖 Generated with Claude Code

needle does not support provide, so these always ended in a 500 that
read as a server fault in the 5xx alerts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@TakGN
TakGN merged commit 3397a46 into main Nov 2, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant