SEO Guide to Creating a Website Uptime Monitor Using Python
It’s Sunday night. You’re lying in bed. Do you know if your website is up?
In this intermediate tutorial, I’ll show you how to create and automate a simple uptime monitor with email notifications using Python. We’ll store historical data in MySQL for later analysis. In addition to detecting whether a site is up or down, we’ll capture basic speed metrics and TLS/SSL verification.
Note: Placeholders exist in some of the code below where you need to fill in the details for your environment.
Table of Contents
Requirements and Assumptions
- Access to a Linux installation (I recommend Ubuntu) and an understanding of basic terminal functions
- Python 3 is installed and you understand basic Python syntax
- Access to a MySQL database, locally or externally
Importing Necessary Modules
Python modules are like libraries in other coding languages. They are collections of premade functions that save you time by avoiding reinvention of the wheel. Most of the Python modules we’re going to use should be preinstalled, but three that aren’t are fake_useragent, yagmail, and mysql.connector. Be sure to read the documentation on yagmail—there’s a small amount of configuration required. To install these, run the following commands in your terminal:
pip3 install mysql.connector
pip3 install fake_useragent
pip3 install yagmail
If you get any errors about other missing modules, you can use the same command above to install them—replace the package name as needed. If a package name isn’t obvious, you can search for it on PyPI.
Creating MySQL Tables
First, we need to set up the database where we’ll store our monitoring data. There are two common ways to access and manage MySQL: via the command line and via the phpMyAdmin GUI in cPanel.
Option 1 – Command Line Terminal:
If you don’t have cPanel or prefer the terminal, follow this guide for logging into MySQL to create the database and user. Then run the SQL statements below to create the tables.
Option 2 – phpMyAdmin:
If you have access to cPanel, you can create the database and user in the MySQL Databases area. After that, open phpMyAdmin (also found in cPanel). Select your database from the list on the left. In the SQL tab at the top, enter the following SQL statement to create the table that will contain the websites you want to monitor. If you already have this table from an earlier guide, you can reuse it.
CREATE TABLE websites (
websiteid int NOT NULL AUTO_INCREMENT,
name varchar(255),
url varchar(255),
PRIMARY KEY (websiteid )
);
At this point you’ll have an empty table for your websites. Populate it with the URLs you want to monitor—typically just the homepage. In phpMyAdmin, select the table and click “Insert” to add each website.
You can also insert website records via SQL as shown below (websiteid is auto-generated):
INSERT INTO websites (name,url) VALUES ("Rocket Clicks","https://www.rocketclicks.com")
Next, we’re ready to create the table for the monitoring data using the SQL statement below:
CREATE TABLE uptime ( scanid int NOT NULL AUTO_INCREMENT, websiteid int(255), date varchar(255), time varchar(255), status_code int(255), speed float(255), cert_valid int(255), PRIMARY KEY (scanid) );
Getting the Script Ready
Fire up your favorite code editor or IDE. I recommend PyCharm for more experienced coders or Thonny for beginners.
Place the following line at the top of the file. It’s called a shebang or hashbang and tells Linux how to execute the file. This is generally optional but required when running from a cronjob, which we will use later. This shebang tells Linux to run the script with Python 3.
#!/usr/bin/python3
First, let’s import the Python modules we’ll use.
## Get today's date from datetime import date ## Get today's time from datetime import datetime ## For header info import requests ## For time delay import time ## To connec to mysql import mysql.connector ## Generate random valid user agents for request from fake_useragent import UserAgent ## For email notifications import yagmail
We’ll work backward slightly by creating two functions: one to handle website requests and another to write to MySQL.
Let’s start with the uptime() function, which requests the site and records the interaction. This function takes three variables: url, name, and header. For the request we use a try/except because we set the verify parameter to True. Setting verify=True validates the TLS certificate and helps avoid scanning malicious URLs. If you set verify=False, the certificate is not validated. If verify=True and the site’s certificate is invalid, requests will raise an exception and the script would stop; the try/except lets us record the event and continue.
def uptime(url,name,header):
try:
response = requests.get(url,headers=header,verify=True)
except:
gettime = "n/a"
status_code_num = "n/a"
cert_valid = "1"
return status_code_num,gettime,cert_valid
Continuing inside the uptime() function, we’ll extract the status_code from the response object. We then test whether the status code indicates success (200). If it is 200, we capture the time it took to download the site’s content using the response’s elapsed timing. If it’s not 200, we skip the speed measurement.
status_code_num = response.status_code
if response.status_code == 200:
gettime = round(response.elapsed.total_seconds(),2)
else:
gettime = 0
return status_code_num,gettime, cert_valid
Now we create the function to store the data returned by uptime(). We generate the date and time here so each record is timestamped for later investigation. Then we build the SQL statement and execute it to insert the record.
def writetolog(websiteid,name,status_code_num,gettime,cert_valid):
now = datetime.now()
today = date.today()
getnow = now.strftime('%H:%M')
getdate = today.strftime('%m/%d/%Y')
mydb = mysql.connector.connect(port="3306", host="HOSTIP",user="USER",password="PASSWORD", database="DATABASE")
cursor = mydb.cursor()
new_scan = "INSERT INTO uptime (websiteid,date,time,code,speed,cert_valid) VALUES ('" + str(websiteid) + "','" + getdate + "','" + getnow + "','" + str(status_code_num) + "','" + str(gettime) + "','" +str(cert_valid)+ "')"
print(new_scan)
cursor.execute(new_scan)
After writing the log record, check whether the status code was 200 or if there is a certificate issue; if not, send a notification that the site is down. Be sure to replace YOUR_GMAIL_ADDRESS with the address you configured for yagmail, and replace TO_EMAIL_ADDRESS with the recipient address for notifications. You can extend the message logic to vary wording for downtime versus certificate issues. In many cases with a severe certificate problem, a browser will alert the user and ask whether to proceed.
if status_code_mum != 200 or cert_valid == 1:
body = "Your website: "+url+"Is DOWN! \n\n"
body += str(response.text.encode('utf8'))
yag = yagmail.SMTP("YOUR_GMAIL_ADDRESS")
yag.send(
to="TO_EMAIL_ADDRESS",
subject=name + " is Down or there is a TLS certificate issue!",
contents=body
)
This is where the script actually begins. Start by fetching the website list from the database—fill in your connection information for the mydb variable. I create another database connection here; if you know whether it’s better to pass mydb and cursor into writetolog, let me know. Then loop through the website records to extract websiteid, name, and url. I included a commented-out line you can use to test downtime with a failing URL.
mydb = mysql.connector.connect(port="3306", host="IP",user="USER",password="PASSWORD", database="DBNAME")
new_scan = "SELECT * FROM websites"
cursor = mydb.cursor()
cursor.execute(new_scan)
records = cursor.fetchall()
for row in records:
websiteid = str(row[0])
name = str(row[1])
url = str(row[2])
#url = "https://expired.badssl.com/" ### For downtime testing
Next, we use fake_useragent to generate a Chrome user-agent, build the header, and call uptime(url, name, header). The function returns the status code, response time, and certificate status, which we then pass to writetolog() to store the record.
ua = UserAgent()
header = {
'User-Agent': ua.chrome
}
status_code_num, gettime, cert_valid = uptime(url,name,header)
writetolog(websiteid,name,status_code_num,gettime, cert_valid)
mydb.close()
Automating the Scan
If your uptime script runs correctly when executed manually, automate it. Linux provides crontab for scheduling recurring tasks. Crontab stores entries that dictate when to execute scripts (time of day, day of week, etc.). To edit your crontab, run:
crontab -e
This typically opens the crontab in the vi editor. On a blank line at the bottom, add the following entry to run the script at midnight every Sunday. Use this cronjob time editor to customize the schedule. Replace the path with your script’s location.
0 0 * * SUN /usr/bin/python3 PATH_TO_SCRIPT/filename.py
To capture a log of each run, redirect output like this (customize the path):
0 0 * * SUN /usr/bin/python3 PATH_TO_SCRIPT/filename.py > PATH_TO_FILE/FILENAME.log 2>&1
Save the crontab file and you’re done. Remember your machine must be powered on at the scheduled time for the cronjob to run.
Conclusion
So there you have it! You can rest easier knowing your websites are monitored. If a site goes down, you’ll receive a notification. I even found an SMS module you can try to send your phone an SMS message. The natural next step is to build a dashboard or other tooling that reads the database for display or analysis. Please follow me on Twitter for feedback and to see interesting extensions of the script. Enjoy!
But wait! We’re not done—coming soon I’ll show how to extend this script to drive an LED monitor board and an LCD screen on a Raspberry Pi so you can visually confirm whether sites are up. Stay tuned!
Uptime Monitor FAQ
How can Python be used to create a website uptime monitor for SEO purposes?
Use Python scripts to implement an uptime monitor that continuously checks website availability to support SEO efforts.
Are there specific Python libraries commonly used for website uptime monitoring?
Python offers libraries such as Requests for HTTP requests and BeautifulSoup for parsing HTML, which can be employed in uptime monitoring workflows.
What considerations should be taken into account when creating a website uptime monitor with Python?
Design robust error handling, choose an appropriate check frequency, and implement timely notifications to promptly address downtime and maintain monitoring effectiveness.
Can Python scripts be scheduled to run periodically for automated website uptime checks?
Yes. Python scripts can be scheduled with tools like cron (on Unix systems) or Task Scheduler (on Windows) to automate periodic uptime checks.
Where can I find a comprehensive guide or documentation for creating a website uptime monitor with Python?
Explore online tutorials, official Python documentation, and SEO-focused resources for step‑by‑step guides and best practices when building a website uptime monitor with Python.
- Evaluate Subreddit Posts in Bulk Using GPT4 Prompting - December 12, 2024
- Calculate Similarity Between Article Elements Using spaCy - November 13, 2024
- Audit URLs for SEO Using ahrefs Backlink API Data - November 11, 2024
