Skip to content

Add async support via optional extra - #50

Open
adarshdigievo wants to merge 5 commits into
masterfrom
adarshdigievo-async-support
Open

adarshdigievo wants to merge 5 commits into
masterfrom
adarshdigievo-async-support

Conversation

@adarshdigievo

@adarshdigievo adarshdigievo commented Nov 15, 2026 •

Copy link
Copy Markdown
Member

AI disclosure: changes made by Fable 5.2 extra and Copilot Fusion Max, via GitHub Copilot Desktop.


Summary

This PR adds Python native async support to the SDK by using httpx2.

I added it as an optional extra without any changes to existing sync client so it will not cause any breaking changes for existing users.

Why not a full migration as disccussed in #43 :

I added serpapi.AsyncClient as an optional extra rather than moving the whole SDK to httpx2. Existing applications can subclass serpapi.Client and customize its requests.Session (for example, by mounting a transport adapter). Replacing that session would break those integrations even if the public method signatures stayed the same. The synchronous client stays on requests; existing users don't need to change their code or install anything new.

Using Async client:

On Python 3.11+, users can install serpapi[async] to use the native httpx2 async client. Search parameters stay the same. Use await for requests, async for for pagination, and async with to close the client. The sync-to-async guide covers request options and transport differences.

The design goal was to keep the behavior as close as possible between sync and async clients.

  • Behavior was tested with 200+ live SerpApi requests covering most SerpApi engines. Both clients produced identical outputs (identical in structure for a few unchached searches) for the same input paramteres.

Type

  • Bug fix
  • Feature
  • Refactor / performance
  • Documentation / examples
  • Tests
  • Maintenance / packaging / CI

Related issue

Closes #43

Affected SDK functionality

  • Python version and operating system: async support on Python 3.11+ with asyncio; tested locally on macOS and in Linux CI.
  • SDK methods or helpers affected: AsyncClient adds search, search archive, Extract, locations, account, and image upload. AsyncSerpResults adds async pagination.
  • SerpApi engine or endpoint involved (if applicable): search across engines, including Extract, plus archive, locations, account, and upload endpoints.
  • Other affected Python versions or environments (confirmed or suspected): Python 3.8–3.10 retain the sync client without the async dependency. CI and release test matrices now use uv to install Python.

Testing

  • AI-executed testing: offline SDK tests, the full SDK suite with an API key, and paired live requests using both clients. No client-specific discrepancy was identified in the completed comparisons.
  • CI results, failures, or tests skipped (with reasons): All eight jobs passed on commit 519f669. Async tests apply only to Python 3.11+; Python 3.8–3.10 exercise sync compatibility.
SDK tests: commands, Python versions, and results
Check Result
python -m pytest tests/test_output_formats.py tests/test_image_upload.py tests/test_extract.py tests/test_timeout.py tests/test_exceptions.py tests/test_async_client.py tests/test_docs_examples.py tests/test_docs_example_runner.py -k 'not live' -q Python 3.11: 138 passed without an API key; 49 deselected by the filter.
python -m pytest tests/test_async_client.py -q 61 passed on each of Python 3.11, 3.14, and 3.15. Covers request encoding, proxies, TLS, timeouts, redirects, pagination, cancellation, and retrying client shutdown.
pytest tests --ignore-glob='tests/test_docs_*.py' -k "not example" Passed in CI on Python 3.8 through 3.15, including authenticated integration tests.
pytest tests/example_search_*_test.py Live engine examples passed in the Python 3.15 CI job.
Full authenticated SDK suite, including documentation examples Python 3.14: 260/263 passed with default cache behavior. A separate run passed 263/263 with no_cache=True only for Google Images examples; both clients returned 100 images on the uncached requests.
Live sync/async comparison: 222 cases across 111 engines

Each client received an independent copy of the same parameters. The comparisons used the complete decoded response body and parsed result, without stripping IDs, sorting arrays, or ignoring fields. All 222 cases were attempted; 220 reached the target through both clients. Two needed prerequisite tokens that were unavailable.

Comparison Result
Engine contract checks 214/222 met the suite's expectations. This measures the API result, separately from whether the two clients agreed.
Final results by case 208 identical live results; 10 different live results with identical archives for the same search IDs; 2 matching SDK errors; 2 targets not reached.
Requests including prerequisites and retries, initial run 267 pairs. All 263 successful pairs matched on HTTP method and URL after removing credentials; 238 had byte-identical bodies and identical parsed results. No transport exception occurred on only one side.
Same-search archives 70/70 exact comparisons across the initial and repeat runs. Fetching the same search ID distinguishes client handling from variation between independent live searches.
Public methods and output formats 14 paired checks covered search and archives in JSON, HTML, and Markdown; pagination; locations; account; Extract scrape, map, crawl, and Markdown; and image upload. 13 matched exactly. Separate uploads differed only in their server-issued image_id.

AI disclosure

  • Model(s): Fable 5.2 extra and Copilot Fusion Max for implementation; GPT-6 Astra for validation; Fable 5.2 and GPT-6.1 Sol for review;
  • Harness/app(s) (e.g., Codex, Claude Code, Cursor): GitHub Copilot Desktop; Claude Code and Copilot CLI for independent review.
  • Used for (e.g., coding, tests, review, documentation): implementation, tests, documentation, live sync/async comparisons, and code review.

Evidence

With serpapi[async] installed, the search parameters are the same as for Client.search():

import asyncio
import os

import serpapi


async def main():
    async with serpapi.AsyncClient(api_key=os.environ["SERPAPI_KEY"]) as client:
        results = await client.search(engine="google", q="coffee")
        print(results["organic_results"][0]["title"])


asyncio.run(main())

adarshdigievo and others added 5 commits November 16, 2026 00:47
Add serpapi.AsyncClient and AsyncSerpResults, backed by httpx2 through the optional serpapi[async] extra on Python 3.11+. The synchronous Client stays on requests.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
ubuntu-latest is now Ubuntu 26.04, where actions/setup-python has no Python 3.8 or 3.9. uv downloads its own builds of every version in the matrix.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Finish aclose() bookkeeping in a done callback, so a cancelled caller no longer leaves closed sessions referenced or an unretrieved task exception. Declare anyio in the test extra, since the async tests use its pytest plugin.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The AsyncClient method docstrings said stream is passed to the underlying HTTP request. The async transport discards it, as the user guide already states.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Assert the expected background close failure and suppress only its matching event-loop diagnostic after caller cancellation.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@adarshdigievo
adarshdigievo marked this pull request as ready for review November 18, 2026 06:42
@adarshdigievo
adarshdigievo requested review from srujana-madd and strzibny and a balanced review from Copilot November 18, 2026 06:42

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The decoding fallback can silently corrupt non-UTF-8 responses whose bytes also form valid UTF-8, altering result fields or API error messages.

1 open issue
What changed in this PR

Adds optional native async support to the SDK through httpx2, while keeping the synchronous client on requests rather than replacing its transport as proposed in #43.

Changes:

  • Adds async SDK methods, pagination, connection management, and transport compatibility handling.
  • Introduces the Python 3.11+ async extra, async tests, and uv-based CI setup.
  • Documents async installation, migration, concurrency, and transport differences.
File Description
tests/​test_exceptions.py Tests optional-dependency errors and lazy imports.
tests/​test_docs_example_runner.py Tests validation of concurrent async examples.
tests/​test_async_client.py Covers async methods, transport behavior, and lifecycle.
tests/​docs_example_support.py Adds async request validation and reporting.
serpapi/​utils.py Adds shared response JSON decoding.
serpapi/​models.py Adds async results and pagination.
serpapi/​http.py Implements async transport and connection management.
serpapi/​exceptions.py Supports error decoding across both transports.
serpapi/​core.py Adds async versions of SDK methods.
README.md Introduces async installation and concurrent searches.
pyproject.toml Adds async and test dependencies.
docs/​user_guide/​threading.md Links asyncio users to the async client.
docs/​user_guide/​request-options.md Clarifies supported methods and transport differences.
docs/​user_guide/​index.md Adds async guide navigation.
docs/​user_guide/​getting-started.md Documents async installation requirements.
docs/​user_guide/​errors-and-timeouts.md Links async exception guidance.
docs/​user_guide/​client-usage.md Adds migration links and clarifies request options.
docs/​user_guide/​async-search-archive.md Distinguishes server-side async jobs from asyncio.
docs/​user_guide/​async-client.md Documents async usage, migration, and limitations.
docs/​reference.md Adds the async shutdown method reference.
docs/​index.md Adds async installation guidance and navigation.
docs/​conf.py Excludes the internal async transport from autodoc.
CONTRIBUTING.md Updates async testing instructions and requirements.
.github/​workflows/​release.yml Uses uv for release-test environment setup.
.github/​workflows/​ci.yml Uses uv for CI environment setup.

Review effort: Balanced

Comment thread serpapi/utils.py
Comment on lines +1 to +19
import json
import os


def api_key_from_environment():
return os.getenv("SERP_API_KEY")


def _response_json(response):
"""Parse the JSON body of a requests or httpx2 response.

httpx2 only decodes UTF-8, UTF-16 and UTF-32 bodies. Like requests, decode
any other body with the response's charset.
"""

try:
return response.json()
except UnicodeDecodeError:
return json.loads(response.text)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed. httpx2's Response.json() is json.loads(self.content), so it decodes the bytes as UTF-8/16/32 whatever charset the response declares, and the .text fallback here only runs after a UnicodeDecodeError. 毛 in GB18030 is C3 AB, which is also valid UTF-8 for ë. With Content-Type: application/json; charset=gb18030, _response_json returns {"q": "ë"}, while json.loads(r.text) returns {"q": "毛"}.

SerpApi sends UTF-8, so this is unlikely in practice. A small fix would be to parse response.text when an httpx2 response declares a charset other than UTF-8 (response.charset_encoding), and keep .json() for everything else.

Confirmed with Opus 5.5 Max.

@srujana-madd

srujana-madd commented Nov 18, 2026 •

Copy link
Copy Markdown

GPT-6-sol model review comments.

I don't think the above issue is for real. Because I see the current constructor setup carries the key across pages.

@strzibny

Copy link
Copy Markdown

I tried to do a first round of review with Opus 5.5 and submitted inline comments, some confirming issues from Copilot.

@strzibny strzibny left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please see inline comments

Comment thread serpapi/http.py
Comment on lines +95 to +107
def filter(self, record):
message = record.getMessage()
redacted = self._API_KEY_IN_URL.sub(r"\1<redacted>", message)
if redacted != message:
record.msg = redacted
record.args = ()
return True


def _protect_httpx2_logs():
logger = logging.getLogger("httpx2")
if not any(isinstance(item, _ApiKeyLogFilter) for item in logger.filters):
logger.addFilter(_ApiKeyLogFilter())

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This filter is added to the process-wide httpx2 logger, so it also rewrites records from other httpx2 clients in the application whenever their URL contains api_key=. With an AsyncClient created, another library's request to https://example.com/other?api_key=THIRD_PARTY_KEY was logged with api_key=<redacted>. Because record.msg becomes the formatted string and record.args becomes (), handlers that read the arguments lose them too.

Could the redaction apply only while this client is sending, and only to the argument that contains the key? For example:

_in_serpapi_request = contextvars.ContextVar("serpapi_request", default=False)


class _ApiKeyLogFilter(logging.Filter):
    _API_KEY_IN_URL = re.compile(r"(?i)([?&]api_key=)[^&\s\"']*")

    def filter(self, record):
        if _in_serpapi_request.get() and isinstance(record.args, tuple):
            record.args = tuple(self._redact(arg) for arg in record.args)
        return True

    def _redact(self, arg):
        text = str(arg)
        redacted = self._API_KEY_IN_URL.sub(r"\1<redacted>", text)
        return redacted if redacted != text else arg

request() would call _in_serpapi_request.set(True) at the start and reset the token in a finally. I ran this with a SerpApi request and an unrelated request in parallel on one httpx2.AsyncClient. Only the SerpApi URL was redacted, and the other record kept its original arguments.

Found with Opus 5.5 Max.

Comment thread serpapi/http.py
Comment on lines +141 to +142
if "api_key" not in params and not api_key_in_data:
params["api_key"] = self.api_key

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This writes api_key into the caller's dict, and search(), search_archive(), locations() and account() also update() the caller's dict with kwargs. When several searches share a base dict, which is a natural way to use asyncio.gather, values leak between calls and between clients:

base = {"engine": "google", "q": "coffee"}
await asyncio.gather(*(client_a.search(base, gl=gl) for gl in ["us", "ca"]))
# base now also contains gl="ca" and client A's api_key
await client_b.search(base)  # sent with gl=ca and client A's api_key

Against a local server, the last request went out with client A's key. Client has the same problem today, but concurrent calls with shared params make it more likely to show up here. extract() already avoids it with params = dict(params) if params is not None else {}. Doing the same in the other methods of both clients, and using params = {**params, "api_key": self.api_key} here, would fix it.

Found with Opus 5.5 Max.

Comment thread serpapi/http.py
Comment on lines +210 to +218
except httpx2.ConnectTimeout as e:
# requests reports connection timeouts as connection errors.
raise _connection_error(e) from e
except httpx2.TimeoutException as e:
raise TimeoutError(str(e) or "The request timed out.") from e
except (
httpx2.NetworkError, httpx2.ProtocolError, httpx2.ProxyError, ssl.SSLError
) as e:
raise _connection_error(e) from e

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few httpx2 errors get past these handlers unwrapped. A response with a corrupt gzip body raises requests.exceptions.ContentDecodingError from Client but a bare httpx2.DecodingError from AsyncClient. That is neither a SerpApiError nor a requests.RequestException, so except blocks carried over from sync code don't catch it. The redirect limit on line 196 raises httpx2.TooManyRedirects, where Client raises requests.exceptions.TooManyRedirects.

The guide covers this ("Other transport failures, such as too many redirects, can still raise httpx2-specific exceptions"), but mapping them takes a few lines and would let you drop that caveat:

except httpx2.TooManyRedirects as e:
    raise requests.exceptions.TooManyRedirects(str(e)) from e
except httpx2.DecodingError as e:
    raise requests.exceptions.ContentDecodingError(str(e)) from e
except httpx2.HTTPError as e:
    raise requests.exceptions.RequestException(str(e)) from e

They need to go after the existing handlers, since the timeout and network errors are also httpx2.HTTPError subclasses.

Found with Opus 5.5 Max.

Comment thread serpapi/models.py
return r.text


class AsyncSerpResults(SerpResults):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isinstance(results, SerpResults) is true for async results, but two inherited methods change shape. next_page() returns a coroutine and yield_pages() returns an async generator, so code written for SerpResults fails far from the cause. if results.next_page(): is always true, since a coroutine is truthy, and ends with a "coroutine was never awaited" warning. for page in results.yield_pages(): raises TypeError: 'async_generator' object is not iterable.

Would you consider moving the shared parts (dict behaviour, next_page_url, pickling, from_http_response) into a private base class and making SerpResults and AsyncSerpResults siblings? Removing the inheritance after a release would be a breaking change, so it's easier to decide now.

Found with Opus 5.5 Max.

Comment thread serpapi/http.py
Comment on lines +403 to +404
if verify is True:
context = ssl.create_default_context(cafile=requests.certs.where())

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Each new session builds a certifi SSLContext here, which is blocking work on the event loop. On my machine it takes about 2.3 ms per context, and AsyncClient() takes about 3.4 ms in total. It runs at construction, for each new proxy/TLS combination, and for every request with an uncached combination once the 16-session cache is full, because those requests get a temporary session.

Building the default certifi context once and reusing it would remove most of that. An SSLContext can be shared by many connections.

The image upload has the same issue. open() in AsyncClient.upload_image and the full file read in _read_body run on the loop (ruff flags the open() as ASYNC230). The guide documents this and images are capped at 500 KB, so it's minor, but asyncio.to_thread would remove the caveat.

Found with Opus 5.5 Max.

Comment thread serpapi/http.py
Comment on lines +232 to +238
self._closed = True
if self._close_task is None:
# Close in a task, so that the sessions still close if the caller is cancelled.
self._close_task = asyncio.create_task(self._close_sessions())
self._close_task.add_done_callback(self._after_close)

await asyncio.shield(self._close_task)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The guide says AsyncClient supports asyncio only, but under Trio the failure comes late. With anyio.run(main, backend="trio"), search() works because httpx2 runs on anyio. Then aclose(), or leaving async with, raises RuntimeError: no running event loop from asyncio.create_task() and leaves the _close_sessions() coroutine un-awaited.

Could __aenter__() and request() check for a running asyncio loop and raise a clear error, such as "serpapi.AsyncClient requires asyncio"? Then the first call fails instead of the cleanup. I'd keep the task plus shield approach for closing. anyio.CancelScope(shield=True) looks like a backend-neutral alternative, but in my test it didn't keep the close running through a plain task.cancel() on asyncio.

Found with Opus 5.5 Max.

Comment thread serpapi/utils.py
Comment on lines +1 to +19
import json
import os


def api_key_from_environment():
return os.getenv("SERP_API_KEY")


def _response_json(response):
"""Parse the JSON body of a requests or httpx2 response.

httpx2 only decodes UTF-8, UTF-16 and UTF-32 bodies. Like requests, decode
any other body with the response's charset.
"""

try:
return response.json()
except UnicodeDecodeError:
return json.loads(response.text)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed. httpx2's Response.json() is json.loads(self.content), so it decodes the bytes as UTF-8/16/32 whatever charset the response declares, and the .text fallback here only runs after a UnicodeDecodeError. 毛 in GB18030 is C3 AB, which is also valid UTF-8 for ë. With Content-Type: application/json; charset=gb18030, _response_json returns {"q": "ë"}, while json.loads(r.text) returns {"q": "毛"}.

SerpApi sends UTF-8, so this is unlikely in practice. A small fix would be to parse response.text when an httpx2 response declares a charset other than UTF-8 (response.charset_encoding), and keep .json() for everything else.

Confirmed with Opus 5.5 Max.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants