The Crawl stats report, tucked in Search Console Settings, shows how Googlebot actually interacts with your server — total requests, download size, response time, and the mix of response codes and file types over the last 90 days. It is a technical, often-ignored report, and for most small sites it can stay ignored. But for large sites, sites with crawl-budget concerns, or anyone diagnosing why new pages take forever to get crawled, it is where the answers live.
What the Report Measures
It reports total crawl requests, total download size, and average response time over 90 days, then breaks requests down by response code, file type, purpose (discovery vs refresh), and Googlebot type.
This is server-side truth about crawling — not indexing. It tells you how much Googlebot is asking for and how your server is answering. A healthy profile is steady crawl volume, fast response times, and a response-code mix dominated by 200s. Anything else is a signal, and it connects directly to the crawlability work that determines whether pages get discovered at all.
Reading Crawl Volume and Trends
Sudden drops in crawl requests can mean Google is throttling because your server slowed down; sudden spikes can mean a crawl trap — infinite URL variations Googlebot is wasting budget on.
Crawl demand tracks your site’s size, health, and how often content changes. A steady decline in requests alongside rising response times usually means the server is struggling and Google is backing off to avoid overloading it — a performance problem, not an SEO one, but it starves crawling. A spike in requests to junk URLs (endless parameter combinations, calendar pages) points to a crawl trap wasting budget that should go to real pages, often the same faceted-navigation URLs behind soft 404s.
Response Codes and Host Status
The response-code breakdown should be mostly 200s; rising 5xx server errors, 429 rate-limits, or DNS/connection failures in host status are the red flags that directly suppress crawling.
Host status summarizes whether Google had trouble reaching your site: robots.txt fetch failures, DNS problems, or server connectivity issues. Any of these can pause crawling entirely. A climbing share of 5xx responses tells Google your server is unreliable and it will crawl less to compensate. Excessive 301s in the mix means Googlebot is spending budget on redirect chains instead of content. Google’s crawl-budget guidance explains how server health shapes crawl rate.
When Crawl Stats Actually Matter
They matter for large sites (100k+ URLs), sites with server-performance issues, and anyone whose new content is crawled too slowly — and they safely do not matter for small, healthy sites.
If you run a few hundred pages on solid hosting and your content gets indexed within days, you never need this report. If you run a huge catalog, or new pages sit in Discovered — currently not indexed for weeks, crawl stats tell you whether the bottleneck is server health, crawl traps, or genuine demand. Match the attention to the scale; this is a large-site diagnostic, not a routine check.
- Crawl stats (in Settings) show 90 days of Googlebot requests, download size, response time, and response-code/file-type breakdowns.
- It measures crawling, not indexing — server-side truth about how Googlebot interacts with your site.
- Declining crawl + rising response time = server throttling; spikes to junk URLs = a crawl trap wasting budget.
- Watch host status and 5xx/429/DNS failures — these directly suppress crawl rate.
- Essential for large or slow-crawled sites; safely ignored by small, healthy ones.
Frequently Asked Questions
Where is the Crawl stats report in Search Console?
It lives under Settings → Crawling → Crawl stats, not in the main left-hand report list. That placement is part of why it is so often overlooked, despite being the best window into Googlebot’s behavior on your site.
Why did my crawl requests suddenly drop?
The most common cause is server slowdown — when response times climb, Google reduces crawl rate to avoid overloading your site. Check the response-time trend and host status; a performance fix usually restores crawl volume. A drop can also follow a large reduction in your URL count.
What is a good average response time?
Lower is better, and the trend matters more than any single number — a rising average is the warning sign. Consistently slow responses cause Google to throttle crawling, so treat response time as a crawl-health metric, not just a user-experience one.
Does crawl budget matter for small sites?
Rarely. Google can comfortably crawl small, healthy sites in full, so crawl budget is mostly a large-site concern. If your pages get indexed within days, you do not have a crawl-budget problem and can leave this report alone.
The Bottom Line
Crawl stats is the report you reach for when crawling itself is the problem — slow discovery, server strain, or budget wasted on junk URLs. For most sites it will only ever confirm that Googlebot is happy. For large or struggling ones, it is the diagnostic that explains what the indexing reports only hint at, and it rounds out the technical side of the Search Console cluster.
Further reading & sources
- Managing crawl budget for large sites — Google Search Central
- Crawl stats report — Google Search Console Help
See how your site actually shows up in AI search. An AI visibility audit maps where you’re cited, where you’re invisible, and what to fix first — in plain English.
Get your AI visibility auditTry the free SEO tools →
Prefer self-serve? The interactive checklists turn guides like this one into a working to-do list.
Keep reading in Google Search Console
Get one email when something genuinely changes
AI search moves fast and most of it is noise. We send one short email when a real shift is worth your time. Unsubscribe anytime.
Published by Plain Intelligence — practical AI SEO, GEO, and technical SEO, documented in plain English. About Plain Intelligence →
↑ Back to Google Search Console · Explore all articles · Free tools & resources · Glossary