feat(waterdata): Add multi-value GET-parameter chunker for OGC API

[thodson-usgs]

Splits multi-value list params across sub-requests so each URL fits the server's ~8 KB byte cap. Without this, get_daily(monitoring_location_id=[300+ sites]) and similar workloads blow past the nginx buffer and the API returns HTTP 414.

The chunker is a decorator wrapped outside the existing @filters.chunked (CQL chunker) on _fetch_once, so list-chunking is the outer loop and CQL-filter-chunking is the inner loop:

@chunking.multi_value_chunked(build_request=_construct_api_requests)   # outer
@filters.chunked(build_request=_construct_api_requests)                # inner
def _fetch_once(args): ...

The two coordinate via a filter-aware probe so the outer planner doesn't reject combinations the stacked chunkers can actually handle.

Planner

_plan_chunks greedy-halves the largest chunk across all dimensions until the worst-case sub-request fits url_limit (URL + body, via _request_bytes, so POST routes size correctly). Cartesian product of per-dim partitions becomes the sub-request set; capped at max_chunks=1000.

Quota guard

Mid-call, the wrapper reads x-ratelimit-remaining after each sub-request. If it drops below quota_safety_floor=50, the wrapper raises QuotaExhausted carrying:

• partial_frame — rows collected so far
• completed_chunks / total_chunks — resume offset
• remaining — last observed quota

Callers can either salvage or resume after the rate-limit window resets, rather than crash into a 429.

Failure modes

Scenario	Result
URL fits without chunking	Decorator passes through; one probe, no overhead
Lists need splitting	Greedy halve → cartesian product of N sub-requests
Filter chunking also needed	Outer mv-loop × inner filter-loop
Filter has clauses bigger than the budget	RequestTooLarge with explicit message
Cartesian product > max_chunks	RequestTooLarge with the actual count
x-ratelimit-remaining drops below floor mid-call	QuotaExhausted with partial_frame, offsets

Public API

dataretrieval.waterdata.chunking.multi_value_chunked   # decorator
dataretrieval.waterdata.chunking.RequestTooLarge       # exception
dataretrieval.waterdata.chunking.QuotaExhausted        # exception

Tests

30 new tests (offline, deterministic fake build_request) covering:

• _filter_aware_probe_args worst-case-clause modelling
• _plan_chunks greedy halving, RequestTooLarge floor, filter-chunker coordination, max_chunks cap, lazy-default reads
• multi_value_chunked pass-through, cartesian-product shape, end-to-end with stacked filter chunker
• QuotaExhausted header parsing, mid-call abort, last-chunk no-abort, zero-floor disable

End-to-end correctness was verified against the live API: per-site cell-for-cell identical output between an unchunked single call and a forced-fan-out chunked call (patched _WATERDATA_URL_BYTE_LIMIT=300 → 30 HTTP sub-requests for the same query).

Known limitations (deferred)

• max_chunks caps the chunk count, not the HTTP request count. A chunked call with 1000 chunks where each paginates 5× = 5000 HTTP requests. Lower max_chunks proportionally for heavy-pagination workloads, or rely on the QuotaExhausted runtime guard.
• No auto-retry on QuotaExhausted. Caller decides whether to wait + resume or fail loudly.
• Sub-requests run sequentially. Parallelism is a follow-up (WIP: perf(waterdata): Optional parallel chunk processing in multi_value_chunked (draft) #278 draft).

Provenance

Supersedes #276 with squashed history. The original PR carried 21 commits across iterative simplify / review-response passes; this one rolls them into a single coherent change atop the now-merged #233 (multi-value GET routing) and #279 (mid-pagination failure surfacing).

Refs #276, #279

[Copilot]

Pull request overview

Adds multi-value GET-parameter chunking to Water Data OGC getters to keep requests under the server’s URL byte limit (preventing HTTP 414), while coordinating with existing CQL filter chunking and adding a quota safety abort mechanism.

Changes:

• Introduces dataretrieval.waterdata.chunking with a greedy chunk planner, RequestTooLarge / QuotaExhausted, and the @multi_value_chunked decorator.
• Wraps utils._fetch_once with @multi_value_chunked outside @filters.chunked, and updates pagination/response metadata aggregation to reflect last headers + cumulative elapsed.
• Adds extensive unit tests for planning/coordination/quota behavior and documents the behavior change in NEWS + get_daily docs.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
dataretrieval/waterdata/chunking.py	New multi-value chunking decorator, planner, and quota guard exceptions.
dataretrieval/waterdata/utils.py	Wires the new chunker into _fetch_once; aggregates paginated response headers/elapsed for accurate metadata and quota inspection.
dataretrieval/waterdata/filters.py	Shares encoding-ratio helper; aggregates chunked responses using first URL + last headers + summed elapsed.
tests/waterdata_test.py	Adds planner/decorator/quota guard tests using a deterministic fake request builder.
dataretrieval/waterdata/api.py	Updates get_daily docstring with a chained-query example relying on transparent chunking.
NEWS.md	Documents the new chunking behavior and metadata semantics changes.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.