You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit d42cde2
Browse filesBrowse the repository at this point in the historyBrowse files
docs: align Extract examples and tests with current API
Document direct Markdown output and current JSON fields, and consolidate
Extract coverage into one test file. Correct the documentation validator
for successful empty Google pagination responses and refresh examples.
Co-authored-by: Codex <codex@openai.com>
Copy file name to clipboardExpand all lines: README.md
+17-19Lines changed: 17 additions & 19 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -47,46 +47,44 @@ Environment variables are a secure, safe, and easy way to manage secrets.
47
47
48
48
### Extract content from a URL
49
49
50
-
The [Extract API](https://serpapi.com/extract-api) supports single-page scraping, URL discovery, and crawling. Call `client.extract()` with a parameter dictionary or keyword arguments. It uses `/search` with `engine=extract` automatically.
50
+
The [Extract API](https://serpapi.com/extract-api) supports single-page scraping, URL discovery, and crawling. Call `client.extract()` with a parameter dictionary or keyword arguments. The client sets `engine="extract"` automatically.
51
+
52
+
Set `output="md"` to get the page's main content as a Markdown string:
JSON responses return `SerpResults`. HTML and Markdown responses return strings. `include_markdown=True` adds the page's Markdown to `text_content["markdown"]` in a JSON response. The module-level `serpapi.extract(url=..., api_key=...)` function accepts the same parameters.
63
+
JSON is the default output format. It returns a dictionary-like `SerpResults` object with page information, structured data, links, and Markdown:
66
64
67
-
To return only the page's Markdown in JSON, combine `include_markdown=True` with `json_restrictor="text_content.markdown"`. This omits all other response sections:
Copy file name to clipboardExpand all lines: docs/index.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -90,7 +90,7 @@ Use the same search parameter names as the [SerpApi API documentation](https://s
90
90
-[Getting Started](user_guide/getting-started.md) covers installation and your first search. [Client Usage](user_guide/client-usage.md) explains how to make requests and read responses.
91
91
-[AI Agents](ai-agents.md) covers search tools, Markdown results, and API references for agents.
92
92
-[Output Formats](user_guide/output-formats.md) explains when to use JSON, Markdown, or HTML.
93
-
-[Extract API](user_guide/extract.md) covers page content, URL discovery, and crawling, including how to return only a page's Markdown in JSON.
93
+
-[Extract API](user_guide/extract.md) covers page content in Markdown or JSON, URL discovery, and crawling.
94
94
-[Migration Guide](user_guide/migrating-from-google-search-results.md) shows how to replace `google-search-results` with this package.
95
95
-[Parameters and Engines](user_guide/parameters-and-engines.md) explains search parameters and how to test them in the [SerpApi Playground](https://serpapi.com/playground).
96
96
- For scripts that collect results, see [pagination](user_guide/pagination.md), [timeouts and errors](user_guide/errors-and-timeouts.md), [request options](user_guide/request-options.md), and [account and locations](user_guide/account-and-locations.md).
Copy file name to clipboardExpand all lines: docs/user_guide/client-usage.md
+4-1Lines changed: 4 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -148,10 +148,13 @@ For one additional page, call `next_page()`:
148
148
next_results = results.next_page()
149
149
```
150
150
151
-
Use `yield_pages()` in a loop to read the current page and fetch more pages. This example stops after ten pages, or earlier if there is no next page:
151
+
Use `yield_pages()` in a loop to read the current page and fetch more pages. This example reads up to ten pages and stops if a page contains an `error` message:
152
152
153
153
```python
154
154
for page_number, page inenumerate(results.yield_pages(max_pages=10), start=1):
155
+
if"error"in page:
156
+
print(page["error"])
157
+
break
155
158
current = page.get("serpapi_pagination", {}).get("current", page_number)
Copy file name to clipboardExpand all lines: docs/user_guide/extract.md
+21-39Lines changed: 21 additions & 39 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -9,9 +9,9 @@ Use `client.extract()` to read content from a public URL. It supports single-pag
9
9
10
10
See the [Extract API documentation](https://serpapi.com/extract-api) for all parameters and response fields.
11
11
12
-
## Get Only the Page Content as JSON
12
+
## Get Page Content as Markdown
13
13
14
-
Set `include_markdown=True` and `json_restrictor="text_content.markdown"` to get the page's Markdown content in a JSON response. This removes metadata and all other response sections.
14
+
Set `output="md"` to get the page's main content as a Markdown string.
15
15
16
16
After [setting your API key](getting-started.md#installation), run:
The response has this structure, with the extracted page content in `markdown`:
35
-
36
-
```json
37
-
{
38
-
"text_content": {
39
-
"markdown": "Page content in Markdown..."
40
-
}
41
-
}
42
-
```
43
-
44
-
The SDK returns a dictionary-like `SerpResults` object. Read `results["text_content"]["markdown"]` to get the plain Markdown string, as shown above. You can pass that string to an AI agent or save it to a file.
45
-
46
-
See [JSON Restrictor](json-restrictor.md) for other ways to select response fields.
31
+
You can pass the returned string to an AI agent or save it to a file.
47
32
48
-
## Get the Full Extract Response
33
+
## Get Structured Data as JSON
49
34
50
-
Leave out `json_restrictor` to receive the available extracted sections, such as page information, headings, links, and text content:
35
+
JSON is the default output format. The SDK returns a dictionary-like `SerpResults` object with page information, meta tags, structured data, links, images, and Markdown. The sections returned depend on the target page.
JSON is the default output format. `include_markdown=True` adds Markdown under `text_content`; the rest of the response remains JSON. The sections returned depend on the target page.
44
+
The `markdown` field contains the page's main content. Set `include_markdown=False` to omit it from the JSON response.
63
45
64
46
The module-level `serpapi.extract(url=..., api_key=...)` function accepts the same parameters.
65
47
66
-
## Markdown and HTML Responses
48
+
## Get HTML
67
49
68
-
Use`output="md"` or `output="html"` to receive a Python string:
50
+
Set`output="html"` to get the retrieved HTML as a string:
html = client.extract(url="https://example.com/", output="html")
53
+
html = client.extract(url="https://serpapi.com/", output="html")
73
54
```
74
55
75
-
For a scraped page, `output="md"` includes search metadata before the page content. Use the [content-only JSON example](#get-only-the-page-content-as-json) when you want the page's Markdown without those extra sections. `output="html"` returns the retrieved HTML.
76
-
77
56
## Discover URLs
78
57
79
58
Set `mode="map"` to discover URLs on a website:
80
59
81
60
```python
82
61
results = client.extract(
83
-
url="https://www.python.org/",
62
+
url="https://serpapi.com/",
84
63
mode="map",
85
64
include_sitemap=False,
86
65
include_subdomains=False,
@@ -98,7 +77,7 @@ Set `mode="crawl"` to follow links and extract pages:
98
77
99
78
```python
100
79
results = client.extract(
101
-
url="https://www.python.org/",
80
+
url="https://serpapi.com/",
102
81
mode="crawl",
103
82
max_pages=2,
104
83
max_depth=1,
@@ -107,17 +86,20 @@ results = client.extract(
107
86
108
87
for page in results["pages"]:
109
88
print(page["url"])
89
+
print(page["markdown"])
110
90
```
111
91
112
-
The starting page has depth `0`. `max_pages` includes the starting page and defaults to `10`. Each successfully extracted page can consume one search credit. Set a page limit for your crawl and allow enough time for the HTTP request to finish. For requests you want to retrieve later, pass `"async": True` in a parameter dictionary and use the [Search Archive](async-search-archive.md).
92
+
Each entry in `results["pages"]` contains the extracted page's data, including its `markdown` field. The starting page has depth `0`. `max_pages` includes the starting page and defaults to `10`. Each successfully extracted page can consume one search credit. Set a page limit for your crawl and allow enough time for the HTTP request to finish.
93
+
94
+
For requests you want to retrieve later, pass `"async": True` in a parameter dictionary and use the [Search Archive](async-search-archive.md).
113
95
114
96
## Timeouts and Content Filters
115
97
116
98
The `timeout=` keyword sets the SDK's HTTP timeout in seconds. Set it above the API fetch timeout, and allow time for all pages when crawling. Put the Extract API's fetch timeout in the parameter dictionary to send it in milliseconds:
:param kwargs: Additional API parameters or HTTP request options, as in :meth:`search`. The ``timeout`` keyword controls the HTTP client timeout in seconds.
72
72
73
73
``mode`` accepts ``scrape`` (default), ``map``, or ``crawl``. ``output`` accepts ``json`` (default), ``html``, or ``md``.
74
+
JSON includes page content in the top-level ``markdown`` field by default,
75
+
or in each ``pages`` entry for crawls. Set ``include_markdown=False`` to omit it.
76
+
For a scraped page, ``output="md"`` returns the page Markdown without search metadata.
74
77
75
78
Crawl mode can consume one search credit per successfully extracted page. Use ``max_pages`` to limit the number of pages.
0 commit comments