Add native Extract API support

CHANGELOG.md

@@ -1,4 +1,5 @@
 # Changelog
+ * [Unreleased] Add native Extract API support via the `extract` method
  * [Unreleased] Apply the `timeout` option to persistent connections
  * [Unreleased] Add Image API support
  * [Unreleased] Add Markdown search and archive output support

README.md

@@ -56,7 +56,7 @@ client.close
 - [Asynchronous searches](./demo/demo_async.rb) for submitting non-blocking jobs and retrieving completed results from the Search Archive API.
 - [Persistent connections and connection pooling](./demo/demo_thread_pool.rb) for reusing HTTP connections across searches.
 - Search results as Ruby hashes with `search`, token-efficient Markdown with `md`, or raw search-engine HTML with `html`.
-- SDK methods for the [Image API](https://serpapi.com/image-api), [Location API](https://serpapi.com/locations-api), [Search Archive API](https://serpapi.com/search-archive-api), and [Account API](https://serpapi.com/account-api).
+- SDK methods for the [Extract API](https://serpapi.com/extract-api), [Image API](https://serpapi.com/image-api), [Location API](https://serpapi.com/locations-api), [Search Archive API](https://serpapi.com/search-archive-api), and [Account API](https://serpapi.com/account-api).
 - Configurable HTTP timeouts and symbolized or string JSON keys.
 
 ## Response formats
@@ -174,6 +174,46 @@ This code shows a simple solution to batch searches asynchronously into a [queue
 
 Here are some examples for some of our most popular APIs. You can find the full list of supported engines and parameters in our [documentation](https://serpapi.com/search-engine-apis).
 
+### Extract API
+
+Turn any public web page into a fully structured JSON document: page information, meta tags, structured data (Open Graph, Twitter Card, JSON-LD, microdata), headings, inline links, images, media, lists, tables, forms, price offers, and normalized text content.
+
+```ruby
+require 'serpapi'
+
+client = SerpApi::Client.new(api_key: ENV['SERPAPI_KEY'])
+results = client.extract(url: 'https://example.com/')
+pp results[:page_information]
+```
+
+The same method exposes two additional modes:
+
+- `map` performs fast same-origin URL discovery via page links, `robots.txt`, and `sitemap.xml`, and returns a `urls` array.
+- `crawl` crawls pages on the same website up to `max_pages` / `max_depth` and returns a fully parsed JSON document for every visited page in a `pages` array.
+
+```ruby
+# discover URLs of a website
+results = client.extract(url: 'https://example.com/', mode: 'map')
+pp results[:urls]
+
+# crawl a website and parse every visited page
+results = client.extract(url: 'https://books.toscrape.com/', mode: 'crawl', max_pages: 3, max_depth: 1)
+pp results[:pages]
+```
+
+To extract a page found in the results, pass its URL to `extract` again. Settings such as `device` or `include_tags` are not carried over, so pass them again if needed:
+
+```ruby
+results = client.extract(url: 'https://example.com/')
+linked_page = client.extract(url: results[:inline_links].first[:link])
+```
+
+ * source code: [demo/demo_extract.rb](https://github.com/serpapi/serpapi-ruby/blob/master/demo/demo_extract.rb)
+
+> **Note:** URLs matching a website already covered by a dedicated SerpApi engine (such as Google, Bing, eBay, or Walmart search pages) are automatically redirected to that engine, and the response follows the matched engine's JSON structure.
+
+[See documentation](https://serpapi.com/extract-api)
+
 ### Google Shopping
 
 Scrape Google Shopping results with product names, prices, ratings, and merchant information.

demo/demo_extract.rb

@@ -0,0 +1,61 @@
+# This script demonstrates the SerpApi Extract API (engine: extract).
+#
+# The Extract API turns any public web page into a fully structured JSON document.
+# The same endpoint exposes three modes:
+# - `scrape` (default): extract a single page into structured JSON
+#   (page information, meta tags, structured data, headings, inline links, text content, ...)
+# - `map`: fast same-origin URL discovery via page links, robots.txt and sitemap.xml
+# - `crawl`: crawl pages on the same website up to `max_pages` / `max_depth`
+#   and return a fully parsed JSON document for every visited page.
+#
+# To extract a page found in the results (e.g. in inline_links), pass its URL
+# to `extract` again.
+#
+# Ensure that the `SERPAPI_KEY` environment variable is set before running this script.
+#
+# doc: https://serpapi.com/extract-api
+
+raise 'SERPAPI_KEY environment variable must be set' if ENV['SERPAPI_KEY'].nil?
+
+require 'pp'
+require 'serpapi'
+
+# client initialization, engine is set per call by the extract method
+client = SerpApi::Client.new(api_key: ENV['SERPAPI_KEY'])
+
+# 1. scrape a single page (default mode)
+puts '--- scrape: https://example.com/'
+results = client.extract(url: 'https://example.com/')
+pp results[:page_information]
+pp results[:text_content]
+
+# 2. map: discover same-origin URLs of a website
+puts '--- map: https://books.toscrape.com/'
+results = client.extract(url: 'https://books.toscrape.com/', mode: 'map', max_pages: 10)
+urls = results[:urls] || []
+puts "discovered #{urls.size} urls"
+pp urls.first(5)
+
+# 3. crawl: parse every visited page up to max_pages / max_depth
+puts '--- crawl: https://books.toscrape.com/'
+results = client.extract(url: 'https://books.toscrape.com/', mode: 'crawl', max_pages: 3, max_depth: 1)
+pages = results[:pages] || []
+pages.each do |page|
+  puts "#{page.dig(:page_information, :title)} - #{page.dig(:page_information, :url)}"
+end
+
+# 4. follow a link: extract the first inline link of https://example.com/
+puts '--- follow: the first inline link of https://example.com/'
+results = client.extract(url: 'https://example.com/')
+link = results[:inline_links]&.first&.dig(:link)
+if link
+  linked_page = client.extract(url: link)
+  pp linked_page[:page_information]
+else
+  puts 'no inline link found to follow'
+end
+
+# close the underlying persistent connection
+client.close
+puts 'done'
+exit 0

lib/serpapi/client.rb

@@ -136,6 +136,51 @@ def md(params = {})
       get('/search', :md, force_output(params, 'md'))
     end
 
+    # Extract structured data from any public web page using the Extract API.
+    #
+    # The Extract API turns a URL into a fully structured JSON document:
+    # page information, meta tags, Open Graph / Twitter Card / Dublin Core,
+    # JSON-LD, microdata, breadcrumbs, headings, inline links / images / videos,
+    # scripts, stylesheets, feeds, lists, tables, forms, sections, price offers,
+    # and normalized text content.
+    #
+    # Modes (via the `mode` parameter):
+    # * `scrape` (default): extract a single page.
+    # * `map`: fast same-origin URL discovery, returns a `urls` array.
+    # * `crawl`: crawl the website up to `max_pages` / `max_depth`, returns a `pages` array.
+    #
+    # **Example:**
+    #
+    # ```ruby
+    # client = SerpApi::Client.new(api_key: ENV['SERPAPI_KEY'])
+    #
+    # # scrape a single page
+    # result = client.extract(url: 'https://example.com/')
+    # pp result[:page_information]
+    #
+    # # crawl a website
+    # result = client.extract(url: 'https://books.toscrape.com/', mode: 'crawl', max_pages: 3, max_depth: 1)
+    # pp result[:pages]
+    # ```
+    #
+    # doc: https://serpapi.com/extract-api
+    #
+    # @param [Hash] params Extract API parameters, must include :url
+    #                (target absolute http(s) URL). Optional parameters:
+    #                mode, max_pages, max_depth, include_subdomains,
+    #                include_sitemap, device, prefetch_delay, timeout,
+    #                include_tags, exclude_tags, include_markdown,
+    #                include_inline_assets.
+    # @return [Hash] extraction results formatted as a Hash.
+    def extract(params = {})
+      raise SerpApiError, "params must be hash, not: #{params.class}" unless params.instance_of?(Hash)
+      raise SerpApiError, 'url parameter is required by the extract API' if params[:url].to_s.empty? && params['url'].to_s.empty?
+
+      # also drop string-keyed engines; 'engine' => nil removes one inherited
+      # from the client defaults, since query compacts nil values
+      search(params.reject { |key, _| key.to_s == 'engine' }.merge('engine' => nil, engine: 'extract'))
+    end
+
     # Get location using Location API
     #
     # example: spec/serpapi/location_api_spec.rb

spec/serpapi/client/example/example_extract_spec.rb

@@ -0,0 +1,27 @@
+require 'spec_helper'
+
+describe 'example: extract' do
+  it 'extracts structured data from a web page' do
+    # initialize the serp api client
+    client = SerpApi::Client.new(api_key: api_key)
+    # extract a single page using the serpapi extract engine
+    results = client.extract(url: 'https://example.com/')
+    expect(results[:page_information]).not_to be_nil, "No page information found! keys available: #{results.keys}"
+
+    # print the output of the response in formatted JSON
+    # pp results[:page_information]
+    # doc: https://serpapi.com/extract-api
+  end
+
+  it 'maps same-origin urls of a website' do
+    client = SerpApi::Client.new(api_key: api_key)
+    results = client.extract(url: 'https://example.com/', mode: 'map')
+    expect(results[:urls]).not_to be_nil, "No urls found! keys available: #{results.keys}"
+  end
+
+  it 'crawls a website and returns parsed pages' do
+    client = SerpApi::Client.new(api_key: api_key)
+    results = client.extract(url: 'https://books.toscrape.com/', mode: 'crawl', max_pages: 2, max_depth: 1)
+    expect(results[:pages]).not_to be_nil, "No pages found! keys available: #{results.keys}"
+  end
+end

spec/serpapi/client/extract_api_spec.rb

@@ -0,0 +1,64 @@
+require 'spec_helper'
+
+describe 'Extract API' do
+  let(:client) { SerpApi::Client.new(api_key: 'dummy_secret_key', persistent: false) }
+
+  # Returns the query parameters sent over HTTP, after merging client defaults.
+  def sent_params
+    http = double
+    response = double(status: 200, body: '{}', headers: { 'Content-Type' => 'application/json' })
+    sent = nil
+    allow(HTTP).to receive(:timeout).and_return(http)
+    allow(http).to receive(:get) do |_url, params:|
+      sent = params
+      response
+    end
+    yield
+    sent
+  end
+
+  describe '#extract' do
+    it 'forces the extract engine and passes the url' do
+      expect(client).to receive(:search).with(hash_including(engine: 'extract', url: 'https://example.com/'))
+      client.extract(url: 'https://example.com/')
+    end
+
+    it 'accepts extra parameters like mode, max_pages and max_depth' do
+      expect(client).to receive(:search).with(
+        hash_including(
+          engine: 'extract',
+          url: 'https://books.toscrape.com/',
+          mode: 'crawl',
+          max_pages: 3,
+          max_depth: 1
+        )
+      )
+      client.extract(url: 'https://books.toscrape.com/', mode: 'crawl', max_pages: 3, max_depth: 1)
+    end
+
+    it 'overrides a conflicting engine parameter' do
+      expect(client).to receive(:search).with(hash_including(engine: 'extract'))
+      client.extract(url: 'https://example.com/', engine: 'google')
+    end
+
+    it 'replaces a string-keyed engine parameter' do
+      params = sent_params { client.extract('url' => 'https://example.com/', 'engine' => 'google') }
+      expect(params.select { |key, _| key.to_s == 'engine' }).to eq(engine: 'extract')
+    end
+
+    it 'replaces a string-keyed engine from the client defaults' do
+      client = SerpApi::Client.new('engine' => 'google', api_key: 'dummy_secret_key', persistent: false)
+      params = sent_params { client.extract(url: 'https://example.com/') }
+      expect(params.select { |key, _| key.to_s == 'engine' }).to eq(engine: 'extract')
+    end
+
+    it 'raises an error when url is missing' do
+      expect { client.extract }.to raise_error(SerpApi::SerpApiError, /url parameter is required/)
+      expect { client.extract(mode: 'map') }.to raise_error(SerpApi::SerpApiError, /url parameter is required/)
+    end
+
+    it 'raises an error when params is not a hash' do
+      expect { client.extract('https://example.com/') }.to raise_error(SerpApi::SerpApiError, /params must be hash/)
+    end
+  end
+end