Extract Google Suggestions API Data for SEO Insights with Python

One of the main tenets of SEO is understanding the search climate for the keywords you are targeting. Understanding what the trends are, what people are searching for, in what volume, and generating ideas from those results. An often overlooked API is Google Suggestions. It’s essentially tapping into Google’s search autocomplete feature.

We can call that feature independent of Google’s search bar and have it return a collection of autocomplete suggestions based on what other people are searching. These are valuable data clues on your audience and can give insight on other keyword options to target.

Amazingly this is one of the easiest APIs to use because there isn’t much to it. No keys, no OAuth, no multiple calls, and few parameters to configure. It almost feels like a side project Google forgot about but keeps the lights on. There isn’t even any official documentation. Let’s dig in using Python before they turn it off!

Key Points

1. One of the main tenets of SEO is understanding the search climate for the keywords you are targeting.
2. Google Suggestions is an often overlooked API that taps into Google‘s search autocomplete feature.
3. It is one of the easiest APIs to use with no keys, no OAuth, no multiple calls, and few parameters to configure.
4. Requires Python 3 and access to a Linux installation (or Google Colab).
5. Install the fake_useragent module for the call header to ward off light abuse detection.
6. Import the requests, json, and user_agent modules.
7. Assign the keyword/keyphrase to a variable with spaces replaced with a plus (+) character or %20.
8. Use the URL http://suggestqueries.google.com/complete/search?output=firefox&q= + keyword to make the call.
9. Create a user agent for the API call to use in the header.
10. Response will be in JSON (or XML if output is set to toolbar).
11. Output can include extra data like google:clientdata, google:suggestrelevance, google:suggesttype and google:verbatimrelevance.
12. Use the response to output cleanly.
13. For deeper analysis, extend the script to go 3 levels deep for each initial single result and then count the frequency.

Requirements and Assumptions

• Python 3 is installed and basic Python syntax understood
• Access to a Linux installation (I recommend Ubuntu) or Google Colab

Starting the Script

First, we install the fake_useragent module to use in our call header. This can help ward off light abuse detection. The API doesn’t have documented usage limits, but Google defenses are always up and if you abuse the API, you may get temporarily or permanently IP blocked. Note if you are using Google Colab, put an exclamation mark at the beginning of pip3, so !pip3.

pip3 install fake_useragent

Next, we import our requests module we will use to make the API call. Then import the JSON module as the response will be in JSON (there is an XML option). Lastly, we import the user_agent module we just installed.

import requests
import json
from fake_useragent import UserAgent

Assign the Keyword

It’s time we assign our keyword/keyphrase to a variable. This is what Google’s suggestion will use to collect the autocomplete data. Since this variable is going into a URL we need to replace spaces with a plus (+) character or you can use %20, doesn’t matter.

keyword = "dog collars"
keyword.replace(" ", "+")

Build the Google Suggestions API Call

The base URL is http://suggestqueries.google.com/complete/search.

Some parameters are as follows:

• client or output = What client to mask as (Firefox, Chrome, Toolbar). This determines the format of the output and if you get extra data. If set to the toolbar you get XML. If set to Firefox or Chrome you get JSON. If you set to Chrome you’ll also get some extra data (I’ll explore that later on).
• hl = Country to use (us, uk , fr…)
• gl = Language to use (en, es, fr…)
• q = Your query

We’re going to keep it super simple and not include HL or GL parameters and set output to Firefox to get JSON response but feel free to experiment.

url = "http://suggestqueries.google.com/complete/search?output=firefox&q=" + keyword

Make the Google Suggestions API Call

Next, we build a user agent for the API call. Then finally use the URL from above and user-agent we just generated to make the call and assign to response to the variable response. verify=false is to disable TLS certificate verification. This is important for requesting untrusted sites, but Google is good and it can cause issues, so we disable it here.

ua = UserAgent()
headers = {"user-agent": ua.chrome}
response = requests.get(url, headers=headers, verify=False)

Process the Google Suggestions API Response

If you choose the output of Firefox you just get this:

["dog collars",["dog collars","dog collars amazon","dog collars etsy","dog collars and leashes","dog collars with name","dog collars personalized","dog collars leather","dog collars near me","dog collars walmart","dog collars for large dogs"],[],{"google:suggestsubtypes":[[433],[],[],[],[],[],[],[],[],[]]}]

If you use output of Chrome you’ll see this JSON response below. It includes some extra data like google:clientdata, google:suggestrelevance, google:suggesttype and google:verbatimrelevance, but I am unsure how to interpret them, so I’ll choose Firefox output. If anyone has insight on those, let me know!

["dog collars",["dog collars amazon","dog collars with name","dog collars etsy","dog collars and leashes","dog collars personalized","dog collars leather","dog collars near me","dog collars walmart"],["","","","","","","",""],[],{"google:clientdata":{"bpc":false,"tlw":false},"google:suggestrelevance":[601,600,555,554,553,552,551,550],"google:suggesttype":["QUERY","QUERY","QUERY","QUERY","QUERY","QUERY","QUERY","QUERY"],"google:verbatimrelevance":1300}]

Output the Response

Since we have our JSON response it’s time to load it into a variable as a list so we can output it cleanly.

suggestions = json.loads(response.text)
for word in suggestions[1]:
  print(word)

Example output

Extra Resources

Colab Notebook
https://colab.research.google.com/drive/18Nta_IO3pi-iu5eYxgGIfjjZ63PIJuKV

Inspiration
https://github.com/Leszek-Sieminski
https://leszeksieminski.me/r/keyword-exploration-googlesuggestqueriesr/

Extend the script
Go 3 levels deep for each initial single result and then count the frequency. Awesome!
https://github.com/omkom-web/scripts-seo/blob/master/gsuggest-distrib.py

Conclusion

With this, you now have the ability to scrape Google suggestions for content ideas programmatically. Now get out there and try it out! Follow me on Twitter and let me know your applications and ideas!

FAQ

What is Google Suggestions?

Google Suggestions is an API that taps into Google‘s search autocomplete feature and returns a collection of autocomplete suggestions based on what other people are searching.

What are some of the parameters needed to make the Google Suggestions API call?

The parameters that are needed are client or output, hl, gl and q. Client or output determines the format of the output and if you get extra data. HL and GL determine the country and language to use. Q is the query.

What is the base URL for the Google Suggestions API call?

The base URL for the Google Suggestions API call is http://suggestqueries.google.com/complete/search.

What is the example output for the Google Suggestions API call?

The example output for the Google Suggestions API call is: “dog collars“,[“dog collars amazon“,”dog collars with name“,”dog collars etsy“,”dog collars and leashes“,”dog collars personalized“,”dog collars leather“,”dog collars near me“,”dog collars walmart“].