Detect Website Technologies with Python & BuiltWith API
For SEO audits, one area you may want to detect and store is the different technologies a website uses. Sure, you can spot-check and run a few console commands, but what if an API could do it all for you? The service BuiltWith provides that capability. BuiltWith offers several useful APIs, but this tutorial focuses on the “Domain API“. This API returns a list of detected technologies and related information such as DNS records. Examples include PHP, GTM, jQuery, Google Fonts, WordPress, IPv6, and hundreds more.
Understanding the technologies in use helps you tackle issues more effectively by giving you a clearer view of the site’s environment. Additionally, BuiltWith returns extra information such as domain ranking scores and detected social accounts. See the full API documentation for complete capabilities.
Note that BuiltWith offers a free web interface for this information. Using the API lets you extract the data for storage, scaling, manipulation, visualization, and automation.
BuiltWith provides a free API tier with limited functionality; most features require credits, including the API used in this tutorial. At the time of writing, pricing is roughly 2,000 API calls for $100.
Table of Contents
Requirements and Assumptions
- Python 3 is installed and basic Python syntax is understood
- Access to a Linux installation (I recommend Ubuntu) or Google Colab
Import Modules
- json: for handling the API response
- requests: for making the API call
import json import requests
Build API Call
Now that the modules are imported, set the domain variable to the site domain you want to check, and use it in the API call stored in url. Several parameters are available; see the documentation here. This tutorial uses three:
- key: your API key for authentication
- liveonly: only send current information, not historical
- lookup: the site domain you want to check
Be sure to add your own API Key to the URL.
domain = "rocketclicks.com" url = "https://api.builtwith.com/v14/api.json?KEY=YOUR_API_KEY&liveonly=yes&LOOKUP="+domain
Next, build the GET request using the url variable and store the response in data. We also create two empty variables to hold technology and social data. You could use a pandas DataFrame, but this tutorial prints to the console.
payload = {}
headers= {}
response = requests.request("GET", url, data = payload)
data = response.json()
technology=""
social=""
Process API Response
We now have the API JSON response in data. The following prints each detected technology’s name, documentation link, and description.
for result in data["Results"][0]["Result"]["Paths"]:
for tech in result["Technologies"]:
technology += tech["Name"] + "\n" + tech["Link"]+ "\n" + tech["Description"] + "\n"
Output
Note: the image below is cropped. The output contained dozens of technologies.
Next, print detected social media accounts for the domain. Each value is the social account URL. We use a try/except because some domains may have no detected social accounts.
try:
for value in data["Results"][0]["Meta"]["Social"]:
social += str(value) + "\n"
except:
social="n/a"
Social Output
Next, print the Majestic Rank, Referring Subnets, Referring IPs, Ahrefs Rank, QuantCast Rank, and Umbrella Rank. Note: these metrics are available only for domains in the top 1M sites (by Majestic).
print("MJRank: " + str(data["Results"][0]["Attributes"]["MJRank"]))
print("RefSN: " + str(data["Results"][0]["Attributes"]["RefSN"]))
print("RefIP: " + str(data["Results"][0]["Attributes"]["RefIP"]))
print("ARank: " + str(data["Results"][0]["Meta"]["ARank"]))
print("QRank: " + str(data["Results"][0]["Meta"]["QRank"]))
print("URank: " + str(data["Results"][0]["Meta"]["Umbrella"]))
Stats Output
As mentioned, the API returns additional data. I highlighted what I found most interesting; to see everything, loop through the full JSON response using the code below.
for key,value in data["Results"][0]["Attributes"].items():
print(str(key) + ":" + str(value))
for key,value in data["Results"][0]["Meta"].items():
print(str(key) + ":" + str(value))
That’s all, folks! As always, be sure to follow me and let me know how this script is working for you and how you are extending it.
Python and BuiltWith API FAQ
How can I use Python to detect web page technologies using the BuiltWith API?
Use Python scripts to call the BuiltWith API, sending requests to detect and retrieve the technologies used by a web page.
What Python libraries are commonly used for making API requests to BuiltWith in order to detect web page technologies?
The requests library is commonly used to make HTTP requests to the BuiltWith API and retrieve technology data from a web page.
What types of technologies can be detected using the BuiltWith API with Python?
The BuiltWith API can identify a variety of technologies used on a web page, including content management systems, web servers, analytics tools, JavaScript frameworks, and more.
Are there any specific considerations or limitations when using the BuiltWith API to detect web page technologies with Python?
Consider the API’s rate limits, handle authentication requirements, and be aware that detection accuracy may vary based on the complexity of the web page.
Where can I find examples and documentation for using Python with the BuiltWith API to detect web page technologies?
Explore the official BuiltWith API documentation for comprehensive guides and examples. Additionally, refer to online tutorials and Python resources for practical demonstrations and implementation details.
- Evaluate Subreddit Posts in Bulk Using GPT4 Prompting - December 12, 2024
- Calculate Similarity Between Article Elements Using spaCy - November 13, 2024
- Audit URLs for SEO Using ahrefs Backlink API Data - November 11, 2024
