Skip to content

Commit d42cde2

Browse files
adarshdigievocodex
andcommitted
docs: align Extract examples and tests with current API
Document direct Markdown output and current JSON fields, and consolidate Extract coverage into one test file. Correct the documentation validator for successful empty Google pagination responses and refresh examples. Co-authored-by: Codex <codex@openai.com>
1 parent de783df commit d42cde2

10 files changed

Lines changed: 220 additions & 336 deletions

File tree

‎README.md‎

Lines changed: 17 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -47,46 +47,44 @@ Environment variables are a secure, safe, and easy way to manage secrets.
4747

4848
### Extract content from a URL
4949

50-
The [Extract API](https://serpapi.com/extract-api) supports single-page scraping, URL discovery, and crawling. Call `client.extract()` with a parameter dictionary or keyword arguments. It uses `/search` with `engine=extract` automatically.
50+
The [Extract API](https://serpapi.com/extract-api) supports single-page scraping, URL discovery, and crawling. Call `client.extract()` with a parameter dictionary or keyword arguments. The client sets `engine="extract"` automatically.
51+
52+
Set `output="md"` to get the page's main content as a Markdown string:
5153

5254
```python
5355
import os
5456
import serpapi
5557

5658
client = serpapi.Client(api_key=os.environ["SERPAPI_KEY"], timeout=60)
57-
results = client.extract(url="https://example.com/", include_markdown=True)
58-
print(results["page_information"]["title"])
59-
print(results["text_content"]["markdown"])
60-
61-
markdown = client.extract(url="https://example.com/", output="md")
62-
html = client.extract(url="https://example.com/", output="html")
59+
markdown = client.extract(url="https://serpapi.com/", output="md")
60+
print(markdown)
6361
```
6462

65-
JSON responses return `SerpResults`. HTML and Markdown responses return strings. `include_markdown=True` adds the page's Markdown to `text_content["markdown"]` in a JSON response. The module-level `serpapi.extract(url=..., api_key=...)` function accepts the same parameters.
63+
JSON is the default output format. It returns a dictionary-like `SerpResults` object with page information, structured data, links, and Markdown:
6664

67-
To return only the page's Markdown in JSON, combine `include_markdown=True` with `json_restrictor="text_content.markdown"`. This omits all other response sections:
65+
```python
66+
results = client.extract(url="https://serpapi.com/")
67+
print(results["page_information"]["title"])
68+
print(results["markdown"])
69+
```
70+
71+
Use `output="html"` to get the retrieved HTML as a string:
6872

6973
```python
70-
results = client.extract(
71-
url="https://example.com/",
72-
include_markdown=True,
73-
json_restrictor="text_content.markdown",
74-
)
75-
print(results["text_content"]["markdown"])
74+
html = client.extract(url="https://serpapi.com/", output="html")
7675
```
7776

7877
Use `mode="map"` to discover URLs or `mode="crawl"` to extract multiple pages:
7978

8079
```python
81-
urls = client.extract(url="https://example.com/", mode="map")
80+
urls = client.extract(url="https://serpapi.com/", mode="map", include_sitemap=False)
8281
pages = client.extract(
83-
url="https://example.com/", mode="crawl", max_pages=2, max_depth=1
82+
url="https://serpapi.com/", mode="crawl", max_pages=2, max_depth=1, timeout=180
8483
)
8584
```
8685

8786
Crawl mode can consume one search credit per successfully extracted page, including the starting page. Set `max_pages` to limit the number of pages.
8887

89-
9088
See the [Extract API guide](https://github.com/serpapi/serpapi-python/blob/master/docs/user_guide/extract.md) for response examples and more options.
9189

9290
### Error handling
@@ -346,7 +344,7 @@ import serpapi
346344
client = serpapi.Client(api_key=os.getenv("API_KEY"))
347345
results = client.search({
348346
'engine': 'google_reverse_image',
349-
'image_url': 'https://i.imgur.com/HBrB8p0.png',
347+
'image_url': 'https://upload.wikimedia.org/wikipedia/commons/6/63/Wikipedia-logo.png',
350348
'max_results': '1',
351349
})
352350
```

‎docs/index.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -90,7 +90,7 @@ Use the same search parameter names as the [SerpApi API documentation](https://s
9090
- [Getting Started](user_guide/getting-started.md) covers installation and your first search. [Client Usage](user_guide/client-usage.md) explains how to make requests and read responses.
9191
- [AI Agents](ai-agents.md) covers search tools, Markdown results, and API references for agents.
9292
- [Output Formats](user_guide/output-formats.md) explains when to use JSON, Markdown, or HTML.
93-
- [Extract API](user_guide/extract.md) covers page content, URL discovery, and crawling, including how to return only a page's Markdown in JSON.
93+
- [Extract API](user_guide/extract.md) covers page content in Markdown or JSON, URL discovery, and crawling.
9494
- [Migration Guide](user_guide/migrating-from-google-search-results.md) shows how to replace `google-search-results` with this package.
9595
- [Parameters and Engines](user_guide/parameters-and-engines.md) explains search parameters and how to test them in the [SerpApi Playground](https://serpapi.com/playground).
9696
- For scripts that collect results, see [pagination](user_guide/pagination.md), [timeouts and errors](user_guide/errors-and-timeouts.md), [request options](user_guide/request-options.md), and [account and locations](user_guide/account-and-locations.md).

‎docs/user_guide/client-usage.md‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -148,10 +148,13 @@ For one additional page, call `next_page()`:
148148
next_results = results.next_page()
149149
```
150150

151-
Use `yield_pages()` in a loop to read the current page and fetch more pages. This example stops after ten pages, or earlier if there is no next page:
151+
Use `yield_pages()` in a loop to read the current page and fetch more pages. This example reads up to ten pages and stops if a page contains an `error` message:
152152

153153
```python
154154
for page_number, page in enumerate(results.yield_pages(max_pages=10), start=1):
155+
if "error" in page:
156+
print(page["error"])
157+
break
155158
current = page.get("serpapi_pagination", {}).get("current", page_number)
156159
print(current)
157160
```

‎docs/user_guide/extract.md‎

Lines changed: 21 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -9,9 +9,9 @@ Use `client.extract()` to read content from a public URL. It supports single-pag
99

1010
See the [Extract API documentation](https://serpapi.com/extract-api) for all parameters and response fields.
1111

12-
## Get Only the Page Content as JSON
12+
## Get Page Content as Markdown
1313

14-
Set `include_markdown=True` and `json_restrictor="text_content.markdown"` to get the page's Markdown content in a JSON response. This removes metadata and all other response sections.
14+
Set `output="md"` to get the page's main content as a Markdown string.
1515

1616
After [setting your API key](getting-started.md#installation), run:
1717

@@ -20,67 +20,46 @@ import os
2020
import serpapi
2121

2222
client = serpapi.Client(api_key=os.environ["SERPAPI_KEY"], timeout=60)
23-
results = client.extract(
24-
url="https://example.com/",
25-
output="json",
26-
include_markdown=True,
27-
json_restrictor="text_content.markdown",
23+
markdown = client.extract(
24+
url="https://serpapi.com/",
25+
output="md",
2826
)
2927

30-
markdown = results["text_content"]["markdown"]
3128
print(markdown)
3229
```
3330

34-
The response has this structure, with the extracted page content in `markdown`:
35-
36-
```json
37-
{
38-
"text_content": {
39-
"markdown": "Page content in Markdown..."
40-
}
41-
}
42-
```
43-
44-
The SDK returns a dictionary-like `SerpResults` object. Read `results["text_content"]["markdown"]` to get the plain Markdown string, as shown above. You can pass that string to an AI agent or save it to a file.
45-
46-
See [JSON Restrictor](json-restrictor.md) for other ways to select response fields.
31+
You can pass the returned string to an AI agent or save it to a file.
4732

48-
## Get the Full Extract Response
33+
## Get Structured Data as JSON
4934

50-
Leave out `json_restrictor` to receive the available extracted sections, such as page information, headings, links, and text content:
35+
JSON is the default output format. The SDK returns a dictionary-like `SerpResults` object with page information, meta tags, structured data, links, images, and Markdown. The sections returned depend on the target page.
5136

5237
```python
53-
results = client.extract(
54-
url="https://example.com/",
55-
include_markdown=True,
56-
)
38+
results = client.extract(url="https://serpapi.com/")
5739

5840
print(results["page_information"]["title"])
59-
print(results["text_content"]["markdown"])
41+
print(results["markdown"])
6042
```
6143

62-
JSON is the default output format. `include_markdown=True` adds Markdown under `text_content`; the rest of the response remains JSON. The sections returned depend on the target page.
44+
The `markdown` field contains the page's main content. Set `include_markdown=False` to omit it from the JSON response.
6345

6446
The module-level `serpapi.extract(url=..., api_key=...)` function accepts the same parameters.
6547

66-
## Markdown and HTML Responses
48+
## Get HTML
6749

68-
Use `output="md"` or `output="html"` to receive a Python string:
50+
Set `output="html"` to get the retrieved HTML as a string:
6951

7052
```python
71-
markdown_response = client.extract(url="https://example.com/", output="md")
72-
html = client.extract(url="https://example.com/", output="html")
53+
html = client.extract(url="https://serpapi.com/", output="html")
7354
```
7455

75-
For a scraped page, `output="md"` includes search metadata before the page content. Use the [content-only JSON example](#get-only-the-page-content-as-json) when you want the page's Markdown without those extra sections. `output="html"` returns the retrieved HTML.
76-
7756
## Discover URLs
7857

7958
Set `mode="map"` to discover URLs on a website:
8059

8160
```python
8261
results = client.extract(
83-
url="https://www.python.org/",
62+
url="https://serpapi.com/",
8463
mode="map",
8564
include_sitemap=False,
8665
include_subdomains=False,
@@ -98,7 +77,7 @@ Set `mode="crawl"` to follow links and extract pages:
9877

9978
```python
10079
results = client.extract(
101-
url="https://www.python.org/",
80+
url="https://serpapi.com/",
10281
mode="crawl",
10382
max_pages=2,
10483
max_depth=1,
@@ -107,17 +86,20 @@ results = client.extract(
10786

10887
for page in results["pages"]:
10988
print(page["url"])
89+
print(page["markdown"])
11090
```
11191

112-
The starting page has depth `0`. `max_pages` includes the starting page and defaults to `10`. Each successfully extracted page can consume one search credit. Set a page limit for your crawl and allow enough time for the HTTP request to finish. For requests you want to retrieve later, pass `"async": True` in a parameter dictionary and use the [Search Archive](async-search-archive.md).
92+
Each entry in `results["pages"]` contains the extracted page's data, including its `markdown` field. The starting page has depth `0`. `max_pages` includes the starting page and defaults to `10`. Each successfully extracted page can consume one search credit. Set a page limit for your crawl and allow enough time for the HTTP request to finish.
93+
94+
For requests you want to retrieve later, pass `"async": True` in a parameter dictionary and use the [Search Archive](async-search-archive.md).
11395

11496
## Timeouts and Content Filters
11597

11698
The `timeout=` keyword sets the SDK's HTTP timeout in seconds. Set it above the API fetch timeout, and allow time for all pages when crawling. Put the Extract API's fetch timeout in the parameter dictionary to send it in milliseconds:
11799

118100
```python
119101
results = client.extract(
120-
{"url": "https://example.com/", "timeout": 30000},
102+
{"url": "https://serpapi.com/", "timeout": 30000},
121103
timeout=60,
122104
)
123105
```

‎serpapi/core.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -71,6 +71,9 @@ def extract(self, params: dict = None, **kwargs):
7171
:param kwargs: Additional API parameters or HTTP request options, as in :meth:`search`. The ``timeout`` keyword controls the HTTP client timeout in seconds.
7272
7373
``mode`` accepts ``scrape`` (default), ``map``, or ``crawl``. ``output`` accepts ``json`` (default), ``html``, or ``md``.
74+
JSON includes page content in the top-level ``markdown`` field by default,
75+
or in each ``pages`` entry for crawls. Set ``include_markdown=False`` to omit it.
76+
For a scraped page, ``output="md"`` returns the page Markdown without search metadata.
7477
7578
Crawl mode can consume one search credit per successfully extracted page. Use ``max_pages`` to limit the number of pages.
7679
"""

‎tests/docs_example_support.py‎

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,18 @@ def validate_response(response, path, params):
4848
if not isinstance(payload, (dict, list)) or not payload:
4949
raise AssertionError("Expected a nonempty JSON object or list")
5050
if isinstance(payload, dict):
51-
if payload.get("error"):
51+
# Google can report an empty final page as Success with an error message.
52+
empty_final_page = (
53+
path in {"/search", "/search.json"}
54+
and params.get("engine") == "google"
55+
and payload.get("search_metadata", {}).get("status") == "Success"
56+
and payload.get("search_information", {}).get("organic_results_state") == "Fully empty"
57+
and payload.get("error") == "Google hasn't returned any results for this query."
58+
and not payload.get("organic_results")
59+
and not payload.get("serpapi_pagination", {}).get("next")
60+
and int(params.get("start", 0)) > 0
61+
)
62+
if payload.get("error") and not empty_final_page:
5263
raise AssertionError(str(payload["error"]))
5364
if payload.get("search_metadata", {}).get("status") == "Error":
5465
raise AssertionError("Search metadata reports an error")

‎tests/test_docs_example_runner.py‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
import json
12
import os
23
import subprocess
34
import sys
@@ -111,6 +112,39 @@ def test_doc_response_check_rejects_json_for_text_output(output):
111112
validate_response(response, "/search", {"output": output})
112113

113114

115+
@pytest.mark.parametrize(
116+
"start, overrides, accepted",
117+
[
118+
pytest.param(70, {}, True, id="empty-final-page"),
119+
pytest.param(0, {}, False, id="empty-first-page"),
120+
pytest.param(70, {"search_metadata": {"status": "Error"}}, False,
121+
id="failed-search"),
122+
pytest.param(70, {"error": "Failed to fetch results"}, False,
123+
id="different-error"),
124+
pytest.param(70, {"serpapi_pagination": {"next": "https://serpapi.com/search?start=80"}},
125+
False, id="not-final-page"),
126+
],
127+
)
128+
def test_doc_response_check_handles_empty_google_pagination(start, overrides, accepted):
129+
payload = {
130+
"search_metadata": {"status": "Success"},
131+
"search_information": {"organic_results_state": "Fully empty"},
132+
"error": "Google hasn't returned any results for this query.",
133+
}
134+
payload.update(overrides)
135+
response = requests.Response()
136+
response.status_code = 200
137+
response.headers["Content-Type"] = "application/json"
138+
response._content = json.dumps(payload).encode()
139+
params = {"engine": "google", "start": str(start)}
140+
141+
if accepted:
142+
validate_response(response, "/search.json", params)
143+
else:
144+
with pytest.raises(AssertionError):
145+
validate_response(response, "/search.json", params)
146+
147+
114148
def test_doc_gate_fails_without_a_key():
115149
env = os.environ.copy()
116150
env.pop("SERPAPI_KEY", None)

‎tests/test_exceptions.py‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
from unittest.mock import Mock
2+
import pytest
23
import requests
34
import serpapi
45

@@ -15,3 +16,18 @@ def test_http_error():
1516
assert http_error.status_code == 401
1617
assert http_error.error == "Invalid API key"
1718
assert http_error.response == mock_response
19+
20+
21+
@pytest.mark.parametrize(
22+
"cause, expected",
23+
[
24+
(requests.exceptions.Timeout, serpapi.TimeoutError),
25+
(requests.exceptions.ConnectionError, serpapi.HTTPConnectionError),
26+
],
27+
)
28+
def test_transport_errors_are_translated(monkeypatch, cause, expected):
29+
client = serpapi.Client(api_key="test-key")
30+
monkeypatch.setattr(client.session, "request", Mock(side_effect=cause("Request failed")))
31+
32+
with pytest.raises(expected):
33+
client.search(engine="google", q="coffee")

0 commit comments

Comments
 (0)