Screaming Frog: Complete Tutorial

By Amit Patel

Screaming Frog SEO Spider is a desktop website crawler that remains the industry standard for hands-on technical SEO audits, more than a decade after its release. This tutorial covers setup, the crawls that matter most in practice, and exactly what the free version gets you versus the paid license.

What It Actually Does

Screaming Frog crawls a website the way a search engine bot does, then reports every URL, status code, title, meta description, heading structure, redirect, and canonical it finds — giving you a complete technical map of a site in one pass, rather than sampling pages manually. It runs locally on your machine (Windows, macOS, or Linux), not as a cloud service, which means very large crawls are limited more by your computer’s memory than by a vendor’s row cap.

It complements rather than replaces platform-level auditing tools like Ahrefs’ Site Audit — Screaming Frog offers far deeper configuration (custom extraction, JavaScript rendering options, arbitrary crawl rules) for one-off deep dives and migrations.

Free vs. Paid: The Real Difference

The free version runs the same crawl engine as the paid license, capped at 500 URLs per crawl, with no crawl saving, no advanced configuration, no JavaScript rendering, and no scheduling — for many small sites, 500 URLs covers the entire site, making the free version genuinely usable, not a crippled trial. The paid license (roughly £199 per year as of 2026, sold per user) removes the URL cap entirely and unlocks JavaScript rendering, custom extraction, scheduling, and integrations with Google Analytics and Search Console.

Check Screaming Frog’s own pricing page for current figures, since pricing and currency display change over time.

Setup and First Crawl

Download the appropriate installer for your OS, install, and run a default “Spider” mode crawl by entering your homepage URL — the default settings crawl HTML, follow internal links, and respect robots.txt, which is the right starting configuration for a first pass. Before crawling a client site or anything you don’t own, confirm you have permission — crawling puts real load on a server, and unannounced heavy crawls of third-party sites can trigger rate-limiting or abuse flags.

Once the crawl completes, the Overview tab and the individual report tabs (Response Codes, Page Titles, Directives) are the fastest way to triage — start with anything 4xx/5xx or redirected, since those are usually the highest-priority fixes.

The Crawls That Matter Most

Four crawl types cover most real technical SEO work: a full-site health crawl (broken links, duplicate titles/metas, redirect chains), a JavaScript-rendering crawl (paid feature) to confirm content renders the way Googlebot sees it, a canonical/hreflang audit for sites with duplicate or international content, and a pre/post-migration crawl comparing URL structures and redirects before and after a site move. Our Technical SEO Checklist and XML sitemap guide pair well with the export data from each.

For sites with tens of thousands of URLs, the free tier’s 500-URL cap becomes a real constraint quickly — that scale of site typically needs the paid license or a cloud-crawling platform’s audit tool instead.

Real Limits

Screaming Frog is desktop software, so very large crawls are bound by your machine’s RAM rather than a fixed vendor limit — an underpowered laptop can struggle on a crawl that a well-specced desktop handles easily. It also requires more manual interpretation than an all-in-one platform’s automated audit scoring; there’s no built-in “priority score” telling you what to fix first, which is a feature, not a bug, for experienced technical SEOs but a steeper learning curve for beginners.

Frequently Asked Questions

The Bottom Line

Screaming Frog remains the default choice for hands-on technical crawls, and its free tier is a genuinely useful starting point rather than a stripped demo. Pair it with a platform like Ahrefs or Semrush for ongoing rank and backlink monitoring, and use it directly for the deep, one-off audits those platforms don’t fully replace.