Technical SEO Audit Checklist: 40 Checks and the Scripts to Run Them

Updated September 2026. This is the technical SEO audit checklist we work through on a site: 40 checks, from crawling and indexing to speed and security. It also includes the two Python scripts we run on every audit, a crawler and a Search Console URL Inspection loop, and four Django problems we find again and again, with fixes.

How we run a technical SEO audit

Most checklists tell you what to check but not how. We audit from three sources of data:

• A crawl of the site, showing what your server actually sends: status codes, redirects, canonicals, titles and headings.
• Google Search Console, showing what Google did with each URL: whether it is indexed, which canonical it chose and which pages earn impressions.
• Field data from real Chrome users (the Core Web Vitals report), showing how fast the site is for visitors rather than in a lab test.

A desktop crawler such as Screaming Frog does the first job well. We use two short Python scripts, shown below the checklist, because they are free and produce CSV files we can join together.

The technical SEO audit checklist

Crawling and robots.txt

1. robots.txt returns 200, or 404 if you have none. Google temporarily treats a 5xx or 429 response as "disallow everything".
2. It doesn't block the CSS, JavaScript or images Google needs to render your pages.
3. Any group for a named crawler, such as Googlebot, repeats every rule that crawler should follow (see the Django section).
4. It points to your XML sitemap with an absolute URL.
5. Important pages are reachable through plain <a href> links, not only through search boxes, forms or JavaScript click handlers.
6. No orphan pages. Every sitemap URL is linked from at least one other page. A crawler's inlinks report shows any that aren't.

Status codes and redirects

7. Every sitemap URL returns 200 directly: no redirects, 404s or server errors.
8. Internal links point at final URLs, not at URLs that redirect.
9. Redirects take one hop, with no chains or loops.
10. Permanent moves use 301 or 308, not 302.
11. Missing pages return a real 404 or 410, not a "page not found" message with a 200 status (a soft 404).
12. One version of every URL. HTTP redirects to HTTPS, www and non-www resolve to one host, and trailing slashes are consistent. Django's APPEND_SLASH handles the last one with a 301.

Indexing and canonicals

13. Every indexable page has exactly one canonical tag, an absolute URL pointing at itself.
14. Paginated pages canonicalise to themselves, not to page 1.
15. Canonical targets return 200 and are indexable, never a redirect, a 404 or a noindex page. Unsure which signal to use? See when to use a canonical tag and when to use a 301 redirect.
16. No accidental noindex in a meta robots tag or an X-Robots-Tag header. A staging setting that leaks into production is the usual cause.
17. Google's chosen canonical matches yours. The URL Inspection API returns both.
18. Sitemap URLs are "Submitted and indexed". A cluster of "Crawled - currently not indexed" usually means thin or near-duplicate pages rather than a technical fault.
19. Filter, sort and tracking parameters don't create crawlable duplicate pages.

XML sitemaps

20. Only canonical, indexable URLs that return 200.
21. lastmod changes only when the content does.
22. Submitted in Search Console and read without errors.
23. No more than 50,000 URLs or 50 MB (uncompressed) per file, with a sitemap index above that.

Titles, headings and structured data

24. A unique <title> on every page, with the main term near the start. Google usually shows about 60 characters.
25. A unique meta description that summarises the page.
26. Exactly one H1 that says what the page is about.
27. Headings in order (H2, then H3), used for structure rather than styling.
28. Structured data passes the Rich Results Test and matches what is visible on the page.
29. A lang attribute on <html> and a viewport meta tag.
30. Images have descriptive alt text and width and height attributes.

JavaScript rendering

If your site is built with React, Vue or another JavaScript framework, these checks matter most. Our comparison of server-side and client-side rendering for SEO explains why.

31. Content, links and meta tags are in the rendered HTML. Compare "View crawled page" in Search Console's URL Inspection with what you see in the browser.
32. Navigation uses <a href> links with real URLs, not click handlers or # fragments.
33. Client-side routes for missing content return a 404 or add noindex, so they don't become soft 404s.
34. JavaScript doesn't change the canonical or robots tag to a different value from the server's HTML.

Speed and mobile

35. Core Web Vitals pass on field data at the 75th percentile: LCP 2.5 seconds or less, INP 200 ms or less, CLS 0.1 or less. Our Core Web Vitals guide covers how to fix each one.
36. The server responds quickly: a time to first byte of 0.8 seconds or less on uncached pages.
37. Text is compressed with Brotli or gzip, and static files are cached for a long time with versioned filenames.
38. The mobile version has the same content, links and structured data as desktop, because Google indexes the mobile version.

HTTPS and security

39. A valid certificate, HSTS and no mixed content. On Django behind a proxy or CDN, set SECURE_PROXY_SSL_HEADER before turning on SECURE_SSL_REDIRECT, or Django can't tell a request was already HTTPS and redirects it in a loop.
40. No manual actions or security issues in Search Console.

Script 1: a crawler that checks status, canonical, title and H1

This crawler starts from your XML sitemap, follows every internal link and image it finds, and records each URL's status code, redirect target, canonical, title, number of H1s and any noindex. It doesn't follow redirects automatically, so every hop appears in the output.

# audit_crawl.py  (pip install requests beautifulsoup4)
# Usage: python audit_crawl.py https://www.example.co.uk
import csv, re, sys
from collections import deque
from urllib.parse import urldefrag, urljoin, urlparse

import requests
from bs4 import BeautifulSoup

START = sys.argv[1].rstrip("/")
HOST = urlparse(START).netloc
session = requests.Session()
session.headers["User-Agent"] = "Mozilla/5.0 (compatible; site-audit)"

sitemap = session.get(START + "/sitemap.xml", timeout=20).text
in_sitemap = set(re.findall(r"<loc>\s*(.*?)\s*</loc>", sitemap))
queue, seen, rows = deque(sorted(in_sitemap) or [START + "/"]), set(), []

while queue and len(seen) < 5000:
    url = queue.popleft()
    if url in seen:
        continue
    seen.add(url)
    r = session.get(url, timeout=20, allow_redirects=False)
    row = {"url": url, "status": r.status_code, "in_sitemap": url in in_sitemap,
           "location": r.headers.get("Location", ""),
           "x_robots": r.headers.get("X-Robots-Tag", "")}
    if r.is_redirect:
        queue.append(urljoin(url, row["location"]))
    elif r.ok and "text/html" in r.headers.get("Content-Type", ""):
        soup = BeautifulSoup(r.text, "html.parser")
        canonical = soup.find("link", rel="canonical")
        robots = soup.find("meta", attrs={"name": "robots"})
        row.update(title=soup.title.get_text(strip=True) if soup.title else "",
                   h1_count=len(soup.find_all("h1")),
                   canonical=canonical.get("href", "") if canonical else "",
                   meta_robots=robots.get("content", "") if robots else "")
        for tag in soup.find_all(["a", "img"]):
            link = urldefrag(urljoin(url, tag.get("href") or tag.get("src") or ""))[0]
            if urlparse(link).netloc == HOST and link not in seen:
                queue.append(link)
    rows.append(row)

fields = ["url", "status", "location", "in_sitemap", "title", "h1_count",
          "canonical", "meta_robots", "x_robots"]
with open("crawl.csv", "w", newline="") as f:
    writer = csv.DictWriter(f, fieldnames=fields)
    writer.writeheader()
    writer.writerows(rows)
print(f"{len(rows)} URLs written to crawl.csv")

On our own site it fetched 145 URLs (116 from the sitemap, plus 29 pages and images found through links) in about 11 seconds, one request at a time. Then filter crawl.csv for sitemap URLs that don't return 200, pages whose canonical isn't their own URL, an h1_count other than 1, duplicate titles and "noindex" in either robots column. The crawler ignores robots.txt and doesn't run JavaScript, which is right for auditing your own server's HTML, so only point it at sites you run.

Script 2: index status for every URL with the Search Console API

The crawl shows what the site serves; the URL Inspection API shows what Google made of it. Create a service account in Google Cloud, enable the Search Console API, download the JSON key, then add the service account's email address as a user on your Search Console property. People often miss that last step, and without it every call fails with a permission error.

# inspect_urls.py  (pip install google-api-python-client google-auth)
import csv

from google.oauth2 import service_account
from googleapiclient.discovery import build

SITE = "sc-domain:example.co.uk"  # or "https://www.example.co.uk/"
creds = service_account.Credentials.from_service_account_file(
    "service-account.json",
    scopes=["https://www.googleapis.com/auth/webmasters.readonly"])
gsc = build("searchconsole", "v1", credentials=creds, cache_discovery=False)

urls = [r["url"] for r in csv.DictReader(open("crawl.csv"))
        if r["in_sitemap"] == "True" and r["status"] == "200"]

with open("index_status.csv", "w", newline="") as f:
    out = csv.writer(f)
    out.writerow(["url", "verdict", "coverage", "google_canonical",
                  "your_canonical", "last_crawl"])
    for url in urls[:2000]:  # 2,000 inspections per property per day
        result = gsc.urlInspection().index().inspect(
            body={"inspectionUrl": url, "siteUrl": SITE}).execute()
        status = result["inspectionResult"]["indexStatusResult"]
        out.writerow([url, status.get("verdict"), status.get("coverageState"),
                      status.get("googleCanonical"), status.get("userCanonical"),
                      status.get("lastCrawlTime")])

Each inspection took about 6.6 seconds in our tests, so 20 URLs took 132 seconds one at a time. With five threads, each with its own API client, the same number took 26 seconds. The per-property limit is 600 inspections a minute, so a few threads stay well within it.

Sort the output by the coverage column. "Discovered - currently not indexed" means Google knows the URL but hasn't crawled it yet, "Crawled - currently not indexed" means it crawled the page and chose not to index it, and "URL is unknown to Google" means it hasn't found it at all. A google_canonical that differs from your_canonical means Google disagrees with your canonical tag, usually because another page is too similar.

Four technical SEO problems we keep finding on Django sites

Paginated pages that canonicalise to page 1

A base template often builds the canonical from request.path, which drops the query string. On /blog/?page=2 that renders a canonical of /blog/, so every archive page claims to be page 1. Google's advice is to give each page in a paginated series its own canonical. Otherwise pages 2 onwards look like duplicates, and posts linked only from them are harder to discover. Add the page number back, and nothing else, so tracking parameters stay out:

<link rel="canonical" href="https://www.example.co.uk{{ request.path }}{% if page_obj.number > 1 %}?page={{ page_obj.number }}{% endif %}">

Link page 1 as /blog/, never /blog/?page=1, so it has one URL.

static() serves nothing when DEBUG is False

The Django documentation offers this line for development, and it often ships to production unchanged:

urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)

django.conf.urls.static.static() returns an empty list when DEBUG is False, so unless your web server handles /media/, every uploaded image returns 404 in production. WhiteNoise won't help, because it serves static files, not uploads. The crawler above catches it, as it records every image's status. The robust fix is to store uploads in object storage such as Amazon S3 or Cloudflare R2 using django-storages. On hosts with a temporary filesystem, local uploads also disappear at the next deploy.

A sitemap lastmod that is always today

Google uses lastmod only when it is consistently accurate. A sitemap that returns timezone.now(), or a model whose auto_now field changes whenever a view counter is saved, tells Google that every page changed today, so it learns to ignore the date. Return a real edit date and update counters without calling save():

from django.contrib.sitemaps import Sitemap
from django.db.models import F

from .models import Post

class PostSitemap(Sitemap):
    def items(self):
        return Post.objects.filter(is_published=True)

    def lastmod(self, post):
        return post.updated_at  # never timezone.now()

# In the detail view: this UPDATE leaves updated_at alone
Post.objects.filter(pk=post.pk).update(views=F("views") + 1)

A Googlebot group that cancels your robots.txt rules

Crawlers obey only the most specific group that matches their name and ignore the rest. In this file, Googlebot may crawl /admin/ and /search/, because its own group allows everything:

User-agent: *
Disallow: /admin/
Disallow: /search/

User-agent: Googlebot
Allow: /

Python's urllib.robotparser picks groups the same way: with this file, can_fetch("Googlebot", ...) returned True for /search/ and the same call for Bingbot returned False. Delete the Googlebot group, or copy every Disallow into it, then confirm in Search Console's robots.txt report.

Turning the checklist into an audit template

An audit is only useful if someone acts on it. We log each failed check as one row, so developers, SEOs and the business owner read the same document:

To rank fixes within a priority, pull impressions per page from the Search Analytics part of the same API and join them to crawl.csv on the URL. A broken canonical on a page with thousands of impressions comes first.

Run the crawl after every major release and the full audit every few months. If you would rather we did it, technical fixes are part of our SEO and marketing work, or book a free call and we'll go through your site with you.