Skip to content

Rate-limit by user and tier in the API, with the WAF as a per-IP backstop #917

Description

@bencap

Summary

Rate limits today are per IP at the load balancer (the WAF), so a signed-in researcher and an anonymous scraper get the same allowance. This issue adds limits in the API keyed on the user or API key, with higher allowances for signed-in users and stricter limits or required sign-in on chosen routes. The WAF stays as a per-IP backstop. It lets the site serve people browsing and working with score sets generously, while scraping the whole corpus through the API stays impractical and bulk users are pointed at the public data dump.

Scope

  1. A limiter in the API keyed on the user for requests signed in or carrying X-API-key, and on the client IP otherwise.
  2. Tiers: anonymous and signed in. Allowances are configuration, not code.
  3. Per-route overrides: a stricter allowance, or sign-in required, on chosen routes. Start with the whole-set routes: the lean variant view, the CSV routes and /variant-details.
  4. Counters shared across API servers, in the store chosen by the caching decision. Postgres is acceptable if request rates allow.
  5. Fail open: if the counter store is unreachable, requests are served and the WAF limits still apply.
  6. Over-limit responses are 429 with Retry-After. On whole-set routes, the body points at the public data dump.
  7. Raise the WAF's per-IP ceiling only as far as the highest tier needs. AWS WAF can't validate an API key, so it never keys on one.
  8. Document the tiers and limits in the Quickstart's Rate Limits section.

Acceptance criteria

  • A signed-in user can make more whole-set requests per window than an anonymous client before getting 429.
  • A route configured to require sign-in returns 401 to anonymous requests.
  • An invented X-API-key is treated as anonymous and doesn't get a fresh allowance.
  • With the counter store unreachable, requests succeed.
  • A 429 from a whole-set route links the public data dump.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    app: awsTask implementation requires AWS environment changesapp: backendTask implementation touches the backend

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions