Robots.txt Guide

1 min read

Robots.txt is a plain-text file that tells crawlers which parts of a site they’re allowed to fetch — a small file with an outsized ability to help or hurt visibility.

Common mistakes

• Accidentally blocking the entire site with a leftover staging rule
• Blocking CSS/JS files a crawler needs to render the page properly
• Forgetting to explicitly allow AI crawlers like GPTBot and PerplexityBot — see AI crawlers explained

A sane baseline

Allow everything except admin/staging paths, link to the XML sitemap, and explicitly welcome the AI crawlers relevant to your visibility goals. See the full technical SEO guide. More in Technical SEO.

Related Reading

Related in Technical SEO:

• XML Sitemap Guide
• Structured Data Guide
• Log File Analysis
• Rendering SEO
• Crawlability Guide

Supporting reading from related clusters:

• Technical SEO Complete Guide
• Generative Engine Optimization Explained
• How to Analyze a Google Core Update

Cornerstone guide: Technical SEO