Repository navigation
Add async support via optional extra - #50
adarshdigievo wants to merge 5 commits into
Conversation
Add serpapi.AsyncClient and AsyncSerpResults, backed by httpx2 through the optional serpapi[async] extra on Python 3.11+. The synchronous Client stays on requests. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
ubuntu-latest is now Ubuntu 26.04, where actions/setup-python has no Python 3.8 or 3.9. uv downloads its own builds of every version in the matrix. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Finish aclose() bookkeeping in a done callback, so a cancelled caller no longer leaves closed sessions referenced or an unretrieved task exception. Declare anyio in the test extra, since the async tests use its pytest plugin. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The AsyncClient method docstrings said stream is passed to the underlying HTTP request. The async transport discards it, as the user guide already states. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Assert the expected background close failure and suppress only its matching event-loop diagnostic after caller cancellation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
The decoding fallback can silently corrupt non-UTF-8 responses whose bytes also form valid UTF-8, altering result fields or API error messages.
1 open issue
What changed in this PR
Adds optional native async support to the SDK through httpx2, while keeping the synchronous client on requests rather than replacing its transport as proposed in #43.
Changes:
- Adds async SDK methods, pagination, connection management, and transport compatibility handling.
- Introduces the Python 3.11+
asyncextra, async tests, and uv-based CI setup. - Documents async installation, migration, concurrency, and transport differences.
| File | Description |
|---|---|
| tests/test_exceptions.py | Tests optional-dependency errors and lazy imports. |
| tests/test_docs_example_runner.py | Tests validation of concurrent async examples. |
| tests/test_async_client.py | Covers async methods, transport behavior, and lifecycle. |
| tests/docs_example_support.py | Adds async request validation and reporting. |
| serpapi/utils.py | Adds shared response JSON decoding. |
| serpapi/models.py | Adds async results and pagination. |
| serpapi/http.py | Implements async transport and connection management. |
| serpapi/exceptions.py | Supports error decoding across both transports. |
| serpapi/core.py | Adds async versions of SDK methods. |
| README.md | Introduces async installation and concurrent searches. |
| pyproject.toml | Adds async and test dependencies. |
| docs/user_guide/threading.md | Links asyncio users to the async client. |
| docs/user_guide/request-options.md | Clarifies supported methods and transport differences. |
| docs/user_guide/index.md | Adds async guide navigation. |
| docs/user_guide/getting-started.md | Documents async installation requirements. |
| docs/user_guide/errors-and-timeouts.md | Links async exception guidance. |
| docs/user_guide/client-usage.md | Adds migration links and clarifies request options. |
| docs/user_guide/async-search-archive.md | Distinguishes server-side async jobs from asyncio. |
| docs/user_guide/async-client.md | Documents async usage, migration, and limitations. |
| docs/reference.md | Adds the async shutdown method reference. |
| docs/index.md | Adds async installation guidance and navigation. |
| docs/conf.py | Excludes the internal async transport from autodoc. |
| CONTRIBUTING.md | Updates async testing instructions and requirements. |
| .github/workflows/release.yml | Uses uv for release-test environment setup. |
| .github/workflows/ci.yml | Uses uv for CI environment setup. |
Review effort: Balanced
| import json | ||
| import os | ||
|
|
||
|
|
||
| def api_key_from_environment(): | ||
| return os.getenv("SERP_API_KEY") | ||
|
|
||
|
|
||
| def _response_json(response): | ||
| """Parse the JSON body of a requests or httpx2 response. | ||
|
|
||
| httpx2 only decodes UTF-8, UTF-16 and UTF-32 bodies. Like requests, decode | ||
| any other body with the response's charset. | ||
| """ | ||
|
|
||
| try: | ||
| return response.json() | ||
| except UnicodeDecodeError: | ||
| return json.loads(response.text) |
There was a problem hiding this comment.
Confirmed. httpx2's Response.json() is json.loads(self.content), so it decodes the bytes as UTF-8/16/32 whatever charset the response declares, and the .text fallback here only runs after a UnicodeDecodeError. 毛 in GB18030 is C3 AB, which is also valid UTF-8 for ë. With Content-Type: application/json; charset=gb18030, _response_json returns {"q": "ë"}, while json.loads(r.text) returns {"q": "毛"}.
SerpApi sends UTF-8, so this is unlikely in practice. A small fix would be to parse response.text when an httpx2 response declares a charset other than UTF-8 (response.charset_encoding), and keep .json() for everything else.
Confirmed with Opus 5.5 Max.
|
GPT-6-sol model review comments.
I don't think the above issue is for real. Because I see the current constructor setup carries the key across pages.
|
|
I tried to do a first round of review with Opus 5.5 and submitted inline comments, some confirming issues from Copilot. |
| def filter(self, record): | ||
| message = record.getMessage() | ||
| redacted = self._API_KEY_IN_URL.sub(r"\1<redacted>", message) | ||
| if redacted != message: | ||
| record.msg = redacted | ||
| record.args = () | ||
| return True | ||
|
|
||
|
|
||
| def _protect_httpx2_logs(): | ||
| logger = logging.getLogger("httpx2") | ||
| if not any(isinstance(item, _ApiKeyLogFilter) for item in logger.filters): | ||
| logger.addFilter(_ApiKeyLogFilter()) |
There was a problem hiding this comment.
This filter is added to the process-wide httpx2 logger, so it also rewrites records from other httpx2 clients in the application whenever their URL contains api_key=. With an AsyncClient created, another library's request to https://example.com/other?api_key=THIRD_PARTY_KEY was logged with api_key=<redacted>. Because record.msg becomes the formatted string and record.args becomes (), handlers that read the arguments lose them too.
Could the redaction apply only while this client is sending, and only to the argument that contains the key? For example:
_in_serpapi_request = contextvars.ContextVar("serpapi_request", default=False)
class _ApiKeyLogFilter(logging.Filter):
_API_KEY_IN_URL = re.compile(r"(?i)([?&]api_key=)[^&\s\"']*")
def filter(self, record):
if _in_serpapi_request.get() and isinstance(record.args, tuple):
record.args = tuple(self._redact(arg) for arg in record.args)
return True
def _redact(self, arg):
text = str(arg)
redacted = self._API_KEY_IN_URL.sub(r"\1<redacted>", text)
return redacted if redacted != text else argrequest() would call _in_serpapi_request.set(True) at the start and reset the token in a finally. I ran this with a SerpApi request and an unrelated request in parallel on one httpx2.AsyncClient. Only the SerpApi URL was redacted, and the other record kept its original arguments.
Found with Opus 5.5 Max.
| if "api_key" not in params and not api_key_in_data: | ||
| params["api_key"] = self.api_key |
There was a problem hiding this comment.
This writes api_key into the caller's dict, and search(), search_archive(), locations() and account() also update() the caller's dict with kwargs. When several searches share a base dict, which is a natural way to use asyncio.gather, values leak between calls and between clients:
base = {"engine": "google", "q": "coffee"}
await asyncio.gather(*(client_a.search(base, gl=gl) for gl in ["us", "ca"]))
# base now also contains gl="ca" and client A's api_key
await client_b.search(base) # sent with gl=ca and client A's api_keyAgainst a local server, the last request went out with client A's key. Client has the same problem today, but concurrent calls with shared params make it more likely to show up here. extract() already avoids it with params = dict(params) if params is not None else {}. Doing the same in the other methods of both clients, and using params = {**params, "api_key": self.api_key} here, would fix it.
Found with Opus 5.5 Max.
| except httpx2.ConnectTimeout as e: | ||
| # requests reports connection timeouts as connection errors. | ||
| raise _connection_error(e) from e | ||
| except httpx2.TimeoutException as e: | ||
| raise TimeoutError(str(e) or "The request timed out.") from e | ||
| except ( | ||
| httpx2.NetworkError, httpx2.ProtocolError, httpx2.ProxyError, ssl.SSLError | ||
| ) as e: | ||
| raise _connection_error(e) from e |
There was a problem hiding this comment.
A few httpx2 errors get past these handlers unwrapped. A response with a corrupt gzip body raises requests.exceptions.ContentDecodingError from Client but a bare httpx2.DecodingError from AsyncClient. That is neither a SerpApiError nor a requests.RequestException, so except blocks carried over from sync code don't catch it. The redirect limit on line 196 raises httpx2.TooManyRedirects, where Client raises requests.exceptions.TooManyRedirects.
The guide covers this ("Other transport failures, such as too many redirects, can still raise httpx2-specific exceptions"), but mapping them takes a few lines and would let you drop that caveat:
except httpx2.TooManyRedirects as e:
raise requests.exceptions.TooManyRedirects(str(e)) from e
except httpx2.DecodingError as e:
raise requests.exceptions.ContentDecodingError(str(e)) from e
except httpx2.HTTPError as e:
raise requests.exceptions.RequestException(str(e)) from eThey need to go after the existing handlers, since the timeout and network errors are also httpx2.HTTPError subclasses.
Found with Opus 5.5 Max.
| return r.text | ||
|
|
||
|
|
||
| class AsyncSerpResults(SerpResults): |
There was a problem hiding this comment.
isinstance(results, SerpResults) is true for async results, but two inherited methods change shape. next_page() returns a coroutine and yield_pages() returns an async generator, so code written for SerpResults fails far from the cause. if results.next_page(): is always true, since a coroutine is truthy, and ends with a "coroutine was never awaited" warning. for page in results.yield_pages(): raises TypeError: 'async_generator' object is not iterable.
Would you consider moving the shared parts (dict behaviour, next_page_url, pickling, from_http_response) into a private base class and making SerpResults and AsyncSerpResults siblings? Removing the inheritance after a release would be a breaking change, so it's easier to decide now.
Found with Opus 5.5 Max.
| if verify is True: | ||
| context = ssl.create_default_context(cafile=requests.certs.where()) |
There was a problem hiding this comment.
Each new session builds a certifi SSLContext here, which is blocking work on the event loop. On my machine it takes about 2.3 ms per context, and AsyncClient() takes about 3.4 ms in total. It runs at construction, for each new proxy/TLS combination, and for every request with an uncached combination once the 16-session cache is full, because those requests get a temporary session.
Building the default certifi context once and reusing it would remove most of that. An SSLContext can be shared by many connections.
The image upload has the same issue. open() in AsyncClient.upload_image and the full file read in _read_body run on the loop (ruff flags the open() as ASYNC230). The guide documents this and images are capped at 500 KB, so it's minor, but asyncio.to_thread would remove the caveat.
Found with Opus 5.5 Max.
| self._closed = True | ||
| if self._close_task is None: | ||
| # Close in a task, so that the sessions still close if the caller is cancelled. | ||
| self._close_task = asyncio.create_task(self._close_sessions()) | ||
| self._close_task.add_done_callback(self._after_close) | ||
|
|
||
| await asyncio.shield(self._close_task) |
There was a problem hiding this comment.
The guide says AsyncClient supports asyncio only, but under Trio the failure comes late. With anyio.run(main, backend="trio"), search() works because httpx2 runs on anyio. Then aclose(), or leaving async with, raises RuntimeError: no running event loop from asyncio.create_task() and leaves the _close_sessions() coroutine un-awaited.
Could __aenter__() and request() check for a running asyncio loop and raise a clear error, such as "serpapi.AsyncClient requires asyncio"? Then the first call fails instead of the cleanup. I'd keep the task plus shield approach for closing. anyio.CancelScope(shield=True) looks like a backend-neutral alternative, but in my test it didn't keep the close running through a plain task.cancel() on asyncio.
Found with Opus 5.5 Max.
| import json | ||
| import os | ||
|
|
||
|
|
||
| def api_key_from_environment(): | ||
| return os.getenv("SERP_API_KEY") | ||
|
|
||
|
|
||
| def _response_json(response): | ||
| """Parse the JSON body of a requests or httpx2 response. | ||
|
|
||
| httpx2 only decodes UTF-8, UTF-16 and UTF-32 bodies. Like requests, decode | ||
| any other body with the response's charset. | ||
| """ | ||
|
|
||
| try: | ||
| return response.json() | ||
| except UnicodeDecodeError: | ||
| return json.loads(response.text) |
There was a problem hiding this comment.
Confirmed. httpx2's Response.json() is json.loads(self.content), so it decodes the bytes as UTF-8/16/32 whatever charset the response declares, and the .text fallback here only runs after a UnicodeDecodeError. 毛 in GB18030 is C3 AB, which is also valid UTF-8 for ë. With Content-Type: application/json; charset=gb18030, _response_json returns {"q": "ë"}, while json.loads(r.text) returns {"q": "毛"}.
SerpApi sends UTF-8, so this is unlikely in practice. A small fix would be to parse response.text when an httpx2 response declares a charset other than UTF-8 (response.charset_encoding), and keep .json() for everything else.
Confirmed with Opus 5.5 Max.

AI disclosure: changes made by Fable 5.2 extra and Copilot Fusion Max, via GitHub Copilot Desktop.
Summary
This PR adds Python native async support to the SDK by using
httpx2.I added it as an optional extra without any changes to existing sync client so it will not cause any breaking changes for existing users.
Why not a full migration as disccussed in #43 :
I added
serpapi.AsyncClientas an optional extra rather than moving the whole SDK tohttpx2. Existing applications can subclassserpapi.Clientand customize itsrequests.Session(for example, by mounting a transport adapter). Replacing that session would break those integrations even if the public method signatures stayed the same. The synchronous client stays onrequests; existing users don't need to change their code or install anything new.Using Async client:
On Python 3.11+, users can install
serpapi[async]to use the nativehttpx2async client. Search parameters stay the same. Useawaitfor requests,async forfor pagination, andasync withto close the client. The sync-to-async guide covers request options and transport differences.The design goal was to keep the behavior as close as possible between sync and async clients.
Type
Related issue
Closes #43
Affected SDK functionality
asyncio; tested locally on macOS and in Linux CI.AsyncClientadds search, search archive, Extract, locations, account, and image upload.AsyncSerpResultsadds async pagination.uvto install Python.Testing
519f669. Async tests apply only to Python 3.11+; Python 3.8–3.10 exercise sync compatibility.SDK tests: commands, Python versions, and results
python -m pytest tests/test_output_formats.py tests/test_image_upload.py tests/test_extract.py tests/test_timeout.py tests/test_exceptions.py tests/test_async_client.py tests/test_docs_examples.py tests/test_docs_example_runner.py -k 'not live' -qpython -m pytest tests/test_async_client.py -qpytest tests --ignore-glob='tests/test_docs_*.py' -k "not example"pytest tests/example_search_*_test.pyno_cache=Trueonly for Google Images examples; both clients returned 100 images on the uncached requests.Live sync/async comparison: 222 cases across 111 engines
Each client received an independent copy of the same parameters. The comparisons used the complete decoded response body and parsed result, without stripping IDs, sorting arrays, or ignoring fields. All 222 cases were attempted; 220 reached the target through both clients. Two needed prerequisite tokens that were unavailable.
image_id.AI disclosure
Evidence
With
serpapi[async]installed, the search parameters are the same as forClient.search():