diff --git a/README.md b/README.md index 8bc208b..46e5fc8 100644 --- a/README.md +++ b/README.md @@ -176,7 +176,7 @@ Here are some examples for some of our most popular APIs. You can find the full ### Extract API -Turn any public web page into a fully structured JSON document: page information, meta tags, structured data (Open Graph, Twitter Card, JSON-LD, microdata), headings, inline links, images, media, lists, tables, forms, price offers, and normalized text content. +Get the main content of any public web page as Markdown, along with page information, meta tags, structured data (Open Graph, Twitter Card, Dublin Core, JSON-LD, microdata), inline links, images, videos, audios, and feeds. ```ruby require 'serpapi' @@ -184,23 +184,41 @@ require 'serpapi' client = SerpApi::Client.new(api_key: ENV['SERPAPI_KEY']) results = client.extract(url: 'https://example.com/') pp results[:page_information] +puts results[:markdown] +``` + +Use `output: 'md'` to get only the page's Markdown as a `String`: + +```ruby +markdown = client.extract(url: 'https://example.com/', output: 'md') ``` The same method exposes two additional modes: -- `map` performs fast same-origin URL discovery via page links, `robots.txt`, and `sitemap.xml`, and returns a `urls` array. -- `crawl` crawls pages on the same website up to `max_pages` / `max_depth` and returns a fully parsed JSON document for every visited page in a `pages` array. +- `map` lists the website's URLs from page links and sitemaps, and returns a `urls` array. Set `include_sitemap: false` to skip the sitemaps and list only URLs linked from the page. Set `include_subdomains: true` to include other subdomains, such as `blog.example.com` when starting from `www.example.com`; this also applies to crawl mode. +- `crawl` follows links on the same website up to `max_pages` / `max_depth` and returns the extracted pages in a `pages` array. Crawl mode consumes one search credit per successfully extracted page, including the starting page. `max_pages` defaults to `10` (maximum `100`), so a crawl without it can consume up to 10 credits. ```ruby -# discover URLs of a website -results = client.extract(url: 'https://example.com/', mode: 'map') +# list the URLs of a website +results = client.extract(url: 'https://books.toscrape.com/', mode: 'map') +pp results[:urls] + +# list only the URLs linked from the page, including other subdomains +results = client.extract(url: 'https://books.toscrape.com/', mode: 'map', include_sitemap: false, include_subdomains: true) pp results[:urls] -# crawl a website and parse every visited page +# crawl a website and extract every visited page results = client.extract(url: 'https://books.toscrape.com/', mode: 'crawl', max_pages: 3, max_depth: 1) pp results[:pages] ``` +A `timeout` passed to `extract` is the Extract API fetch timeout in milliseconds. The client `timeout` is the HTTP timeout in seconds for the whole request, so raise it as well for slow pages or crawls: + +```ruby +client = SerpApi::Client.new(api_key: ENV['SERPAPI_KEY'], timeout: 180) +results = client.extract(url: 'https://example.com/', timeout: 60_000) +``` + To extract a page found in the results, pass its URL to `extract` again. Settings such as `device` or `include_tags` are not carried over, so pass them again if needed: ```ruby @@ -210,7 +228,7 @@ linked_page = client.extract(url: results[:inline_links].first[:link]) * source code: [demo/demo_extract.rb](https://github.com/serpapi/serpapi-ruby/blob/master/demo/demo_extract.rb) -> **Note:** URLs matching a website already covered by a dedicated SerpApi engine (such as Google, Bing, eBay, or Walmart search pages) are automatically redirected to that engine, and the response follows the matched engine's JSON structure. +> **Note:** URLs of pages covered by a dedicated SerpApi engine (such as Google, Bing, eBay, or Walmart search pages) are redirected to that engine and return its results. [See documentation](https://serpapi.com/extract-api) diff --git a/demo/demo_extract.rb b/demo/demo_extract.rb index 44676ab..eb19f2a 100644 --- a/demo/demo_extract.rb +++ b/demo/demo_extract.rb @@ -1,12 +1,12 @@ # This script demonstrates the SerpApi Extract API (engine: extract). # -# The Extract API turns any public web page into a fully structured JSON document. +# The Extract API returns the main content of any public web page as Markdown, +# along with page information, meta tags, structured data and inline links. # The same endpoint exposes three modes: -# - `scrape` (default): extract a single page into structured JSON -# (page information, meta tags, structured data, headings, inline links, text content, ...) -# - `map`: fast same-origin URL discovery via page links, robots.txt and sitemap.xml -# - `crawl`: crawl pages on the same website up to `max_pages` / `max_depth` -# and return a fully parsed JSON document for every visited page. +# - `scrape` (default): extract a single page +# - `map`: list the website's URLs from page links and sitemaps +# - `crawl`: follow links on the same website up to `max_pages` / `max_depth` +# and extract every visited page (one search credit per extracted page) # # To extract a page found in the results (e.g. in inline_links), pass its URL # to `extract` again. @@ -27,24 +27,28 @@ puts '--- scrape: https://example.com/' results = client.extract(url: 'https://example.com/') pp results[:page_information] -pp results[:text_content] +puts results[:markdown] -# 2. map: discover same-origin URLs of a website +# 2. get only the page's Markdown as a String +puts '--- markdown output: https://example.com/' +puts client.extract(url: 'https://example.com/', output: 'md') + +# 3. map: list the URLs of a website puts '--- map: https://books.toscrape.com/' -results = client.extract(url: 'https://books.toscrape.com/', mode: 'map', max_pages: 10) +results = client.extract(url: 'https://books.toscrape.com/', mode: 'map') urls = results[:urls] || [] puts "discovered #{urls.size} urls" pp urls.first(5) -# 3. crawl: parse every visited page up to max_pages / max_depth +# 4. crawl: extract every visited page up to max_pages / max_depth puts '--- crawl: https://books.toscrape.com/' results = client.extract(url: 'https://books.toscrape.com/', mode: 'crawl', max_pages: 3, max_depth: 1) pages = results[:pages] || [] pages.each do |page| - puts "#{page.dig(:page_information, :title)} - #{page.dig(:page_information, :url)}" + puts "#{page.dig(:page_information, :title)} - #{page[:url]}" end -# 4. follow a link: extract the first inline link of https://example.com/ +# 5. follow a link: extract the first inline link of https://example.com/ puts '--- follow: the first inline link of https://example.com/' results = client.extract(url: 'https://example.com/') link = results[:inline_links]&.first&.dig(:link) diff --git a/lib/serpapi/client.rb b/lib/serpapi/client.rb index b8ae22a..14eeb94 100644 --- a/lib/serpapi/client.rb +++ b/lib/serpapi/client.rb @@ -136,18 +136,25 @@ def md(params = {}) get('/search', :md, force_output(params, 'md')) end - # Extract structured data from any public web page using the Extract API. + # Extract the content of any public web page using the Extract API. # - # The Extract API turns a URL into a fully structured JSON document: + # The Extract API returns the page's main content as Markdown, along with # page information, meta tags, Open Graph / Twitter Card / Dublin Core, - # JSON-LD, microdata, breadcrumbs, headings, inline links / images / videos, - # scripts, stylesheets, feeds, lists, tables, forms, sections, price offers, - # and normalized text content. + # JSON-LD, microdata, inline links / images / videos / audios, and feeds. + # URLs covered by a dedicated SerpApi engine (such as Google or Bing search + # pages) are redirected to that engine and return its results. # # Modes (via the `mode` parameter): # * `scrape` (default): extract a single page. - # * `map`: fast same-origin URL discovery, returns a `urls` array. - # * `crawl`: crawl the website up to `max_pages` / `max_depth`, returns a `pages` array. + # * `map`: list the website's URLs from page links and sitemaps, returns a `urls` array. + # Set `include_sitemap: false` to list only URLs linked from the page. + # Set `include_subdomains: true` to include other subdomains (also in `crawl` mode). + # * `crawl`: follow links on the same website up to `max_pages` / `max_depth`, + # returns a `pages` array. Consumes one search credit per extracted page. + # `max_pages` defaults to 10 (maximum 100), including the starting page. + # + # The `timeout` parameter is the Extract API fetch timeout in milliseconds. + # The client HTTP timeout is set in seconds with `SerpApi::Client.new(timeout: ...)`. # # **Example:** # @@ -156,7 +163,10 @@ def md(params = {}) # # # scrape a single page # result = client.extract(url: 'https://example.com/') - # pp result[:page_information] + # puts result[:markdown] + # + # # get only the page's Markdown as a String + # markdown = client.extract(url: 'https://example.com/', output: 'md') # # # crawl a website # result = client.extract(url: 'https://books.toscrape.com/', mode: 'crawl', max_pages: 3, max_depth: 1) @@ -168,10 +178,10 @@ def md(params = {}) # @param [Hash] params Extract API parameters, must include :url # (target absolute http(s) URL). Optional parameters: # mode, max_pages, max_depth, include_subdomains, - # include_sitemap, device, prefetch_delay, timeout, - # include_tags, exclude_tags, include_markdown, - # include_inline_assets. - # @return [Hash] extraction results formatted as a Hash. + # include_sitemap, prefetch_delay, timeout, include_tags, + # exclude_tags, include_markdown, device, no_cache, async, + # zero_trace, output, json_restrictor. + # @return [Hash|String] extraction results as a Hash, or a String for `output: 'md'` or `'html'`. def extract(params = {}) raise SerpApiError, "params must be hash, not: #{params.class}" unless params.instance_of?(Hash) raise SerpApiError, 'url parameter is required by the extract API' if params[:url].to_s.empty? && params['url'].to_s.empty? diff --git a/spec/serpapi/client/example/example_extract_spec.rb b/spec/serpapi/client/example/example_extract_spec.rb index 13111a5..b1764a3 100644 --- a/spec/serpapi/client/example/example_extract_spec.rb +++ b/spec/serpapi/client/example/example_extract_spec.rb @@ -7,12 +7,20 @@ # extract a single page using the serpapi extract engine results = client.extract(url: 'https://example.com/') expect(results[:page_information]).not_to be_nil, "No page information found! keys available: #{results.keys}" + expect(results[:markdown]).not_to be_nil, "No markdown found! keys available: #{results.keys}" # print the output of the response in formatted JSON # pp results[:page_information] # doc: https://serpapi.com/extract-api end + it 'extracts the markdown of a web page' do + client = SerpApi::Client.new(api_key: api_key) + markdown = client.extract(url: 'https://example.com/', output: 'md') + expect(markdown).to be_a(String) + expect(markdown).not_to be_empty + end + it 'maps same-origin urls of a website' do client = SerpApi::Client.new(api_key: api_key) results = client.extract(url: 'https://books.toscrape.com/', mode: 'map')