Limited JSON API for Google searches using Programmable Search Engine#

I figured out how to use a JSON API to run a very limited Google search today in a legit, non-screen-scraper way.

Google offer a product called Programmable Search Engine, which used to be called Google Custom Search.

It’s intended for creating a search engine for your own site, by restricting results to specific domains - but when you create one you can opt to search the whole web instead.

You can then use their JSON API to run searches.

It’s quite limited:

Maximum of 10 results per page, and you can only paginate up to 10 times so 100 results total
100 search queries per day for free, then $5 per 1000 queries up to 10,000 a day, then you have to switch to a separate product that only works for up to 10 domains.

But it works! And it’s pretty easy to get running.

First, create a new Programmable Search Engine from the dashboard. The create page is pretty straight-forward:

Screenshot of the create form - you basically just need to give it a name and solve a captcha

Now get an API key - I used the button in the middle of the API documentation:

A nice blue button for creating an API key

You need the “Search engine ID” from the dashboard - mine was 84ec3c54dca9646ff.

And that’s it! You can combine the API key and search engine ID to run searches:

1
https://www.googleapis.com/customsearch/v1?key=API-KEY
2
  &cx=84ec3c54dca9646ff
3
  &q=SEARCH-TERM

It seems to support a lot of the same search filters as Google. I tried using this, URL-encoded, and seemed to get the results I wanted:

1
"powered by datasette" -site:github.com -site:simonwillison.net -site:datasette.io -site:pypi.org

The results come back as JSON that looks like this (truncated after the first result):

1
{
2
  "kind": "customsearch#search",
3
  "url": {
4
    "type": "application/json",
5
    "template": "https://www.googleapis.com/customsearch/v1?q={searchTerms}&num={count?}&start={startIndex?}&lr={language?}&safe={safe?}&cx={cx?}&sort={sort?}&filter={filter?}&gl={gl?}&cr={cr?}&googlehost={googleHost?}&c2coff={disableCnTwTranslation?}&hq={hq?}&hl={hl?}&siteSearch={siteSearch?}&siteSearchFilter={siteSearchFilter?}&exactTerms={exactTerms?}&excludeTerms={excludeTerms?}&linkSite={linkSite?}&orTerms={orTerms?}&relatedSite={relatedSite?}&dateRestrict={dateRestrict?}&lowRange={lowRange?}&highRange={highRange?}&searchType={searchType}&fileType={fileType?}&rights={rights?}&imgSize={imgSize?}&imgType={imgType?}&imgColorType={imgColorType?}&imgDominantColor={imgDominantColor?}&alt=json"
6
  },
7
  "queries": {
8
    "request": [
9
      {
10
        "title": "Google Custom Search - \"powered by datasette\" -site:github.com -site:simonwillison.net -site:datasette.io -site:pypi.org",
11
        "totalResults": "65200",
12
        "searchTerms": "\"powered by datasette\" -site:github.com -site:simonwillison.net -site:datasette.io -site:pypi.org",
13
        "count": 10,
14
        "startIndex": 1,
15
        "inputEncoding": "utf8",
16
        "outputEncoding": "utf8",
17
        "safe": "off",
18
        "cx": "84ec3c54dca9646ff"
19
      }
20
    ],
21
    "nextPage": [
22
      {
23
        "title": "Google Custom Search - \"powered by datasette\" -site:github.com -site:simonwillison.net -site:datasette.io -site:pypi.org",
24
        "totalResults": "65200",
25
        "searchTerms": "\"powered by datasette\" -site:github.com -site:simonwillison.net -site:datasette.io -site:pypi.org",
26
        "count": 10,
27
        "startIndex": 11,
28
        "inputEncoding": "utf8",
29
        "outputEncoding": "utf8",
30
        "safe": "off",
31
        "cx": "84ec3c54dca9646ff"
32
      }
33
    ]
34
  },
35
  "context": {
36
    "title": "The whole web"
37
  },
38
  "searchInformation": {
39
    "searchTime": 0.25516,
40
    "formattedSearchTime": "0.26",
41
    "totalResults": "65200",
42
    "formattedTotalResults": "65,200"
43
  },
44
  "items": [
45
    {
46
      "kind": "customsearch#result",
47
      "title": "hhs",
48
      "htmlTitle": "hhs",
49
      "link": "https://hhscovid.publicaccountability.org/hhs",
50
      "displayLink": "hhscovid.publicaccountability.org",
51
      "snippet": "Powered by Datasette · Queries took 5.536ms · Data source: U.S. Department of Health & Human Services · Home · Name Search · Dataset Search · Browse Datasets.",
52
      "htmlSnippet": "<b>Powered by Datasette</b> · Queries took 5.536ms · Data source: U.S. Department of Health &amp; Human Services &middot; Home &middot; Name Search &middot; Dataset Search &middot; Browse Datasets.",
53
      "cacheId": "QbpCTHbMliYJ",
54
      "formattedUrl": "https://hhscovid.publicaccountability.org/hhs",
55
      "htmlFormattedUrl": "https://hhscovid.publicaccountability.org/hhs",
56
      "pagemap": {
57
        "metatags": [
58
          {
59
            "viewport": "width=device-width, initial-scale=1, shrink-to-fit=no"
60
          }
61
        ]
62
      }
63
    }

As a bonus, you can pipe results into a SQLite database using sqlite-utils like this:

1
curl 'https://www.googleapis.com/customsearch...' | \
2
  jq .items | sqlite-utils insert /tmp/search.db search -