Crawl Control

Track 3 · Lesson 1 of 6 · about 15 minutes

Crawl control

A crawler arrives at your site with finite patience. Crawl control is deciding, on purpose, where that patience gets spent — instead of letting your URL structure decide by accident.

robots.txt is a request line, not a lock

robots.txt tells cooperating bots which paths they may request. Three properties define its correct use. It controls crawling only — a disallowed URL can still be indexed from external signals, just with no snippet. It is voluntary — major crawlers comply by convention, nothing enforces it. And it is blunt — path prefixes and wildcards, nothing conditional. Use it to keep crawlers out of infinite spaces (search results, filters, carts), never as a privacy tool and never in combination with noindex on the same URL — the block prevents the noindex from ever being read.

Sitemaps are a curated invitation

The XML sitemap is the inverse instrument: instead of fencing off, it nominates. The rules from Track 1 hold — canonical, indexable, current URLs only — and one addition matters at scale: lastmod, kept truthful. Google has said it uses lastmod when it has proven reliable; a sitemap that stamps today’s date on everything teaches the crawler to ignore yours. Segmenting large sites into multiple sitemaps also turns the Search Console sitemap report into a diagnostic: you can see which section’s URLs are being ignored.

Crawl budget is real, above a certain size

For a few hundred pages, budget anxiety is misplaced — discovery problems at that scale are internal-linking problems. The calculus changes when URL space explodes: faceted navigation, session parameters, calendar pagination, infinite scroll variants. The tell-tale is in your logs or the Crawl Stats report: thousands of hits on parameter combinations, sparse hits on the pages you care about. The fix is structural — collapse the space. Decide which few facet combinations deserve to exist as crawlable URLs, serve clean versions of those, and disallow the generator patterns producing the rest.

The audit that takes twenty minutes

Fetch your own robots.txt and read it line by line — most sites carry rules nobody remembers writing. Check your sitemap for redirects, 404s and noindexed URLs (a crawler like Screaming Frog does this in one pass). Then open Crawl Stats in Search Console and look at what Googlebot actually spent its last ninety days fetching. The gap between what you meant and what is happening is usually visible in minutes.

Going deeper: robots.txt explained · robots.txt generator · sitemap generator.