Skip to content

Add pandas 3x, legacy xfails, and samples API key#221

Merged
thodson-usgs merged 4 commits intoDOI-USGS:mainfrom
thodson-usgs:pandas-3.0
Mar 19, 2026
Merged

Add pandas 3x, legacy xfails, and samples API key#221
thodson-usgs merged 4 commits intoDOI-USGS:mainfrom
thodson-usgs:pandas-3.0

Conversation

@thodson-usgs
Copy link
Collaborator

@thodson-usgs thodson-usgs commented Mar 19, 2026

PR: Support for Pandas 3.0 and NWIS Service Migrations

Summary

This PR adds support for Pandas 3.0, handles the recent retirement of legacy NWIS RDB services (redirecting to HTML), and fixes a critical bug in API Key (PAT) header implementation.

Key Changes

1. Dependencies and Core Logic

  • pyproject.toml: Updated pandas constraint to pandas>=2.0.0,<4.0.0.
  • dataretrieval/utils.py: Updated pd.to_datetime to use format="mixed". This ensures compatibility with Pandas 3.0's strict ISO8601 validation while handling datasets with mixed date/time formats.

2. NWIS Module Enhancements

  • gwlevels Migration: Re-routed the get_gwlevels service from the legacy waterdata URL (which now redirects to an HTML UI) to the modern waterservices.usgs.gov RDB endpoint.
  • Robustness: Enhanced _read_rdb to gracefully handle unexpected HTML responses from decommissioned services.
  • Backward Compatibility: Implemented a fallback for the parameter_cd column and other schema differences to ensure existing user code remains functional even as backend services evolve.

3. API Key (PAT) Fix

  • dataretrieval/waterdata/api.py: Patched get_samples and code lookup functions to correctly include the X-Api-Key header when API_USGS_PAT is provided, resolving persistent 429 Rate Limit errors.

4. Test Suite Maintenance

  • tests/waterdata_test.py: Updated datetime resolution assertions to support both nanosecond (ns) and microsecond (us) Defaults (Pandas 3.0 uses us).
  • tests/nwis_test.py: Marked legacy services that have been retired from RDB output (like measurements and peaks) with XFAIL.

Verification Results

Verified with both Pandas 3.0.1 and Pandas 2.3.3:

  • Total Tests: 129
  • Passed: 120
  • Expected Failures (XFAIL): 7 (Retired legacy endpoints)
  • Skips: 2

@thodson-usgs thodson-usgs self-assigned this Mar 19, 2026
"""
if "<html>" in rdb.lower() or "<!doctype html>" in rdb.lower():
raise ValueError(
"Received HTML response instead of RDB. This often indicates "
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this and plan to steal it in R!

@thodson-usgs thodson-usgs merged commit 2c775e4 into DOI-USGS:main Mar 19, 2026
8 checks passed
@thodson-usgs thodson-usgs deleted the pandas-3.0 branch March 19, 2026 17:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants