Robots.txt Guide

By Amit Patel

One misplaced line in a text file can hide an entire website from Google. Robots.txt is the file that tells crawlers which parts of your site they may request — a powerful but blunt instrument that is as easy to misuse as it is to use. In 2026 it also governs how AI crawlers access your content, adding new stakes. This guide explains what robots.txt does, how to write it correctly, and the mistakes that quietly cost sites their rankings.

What Robots.txt Does and Does Not Do

Robots.txt tells compliant crawlers which URLs they may or may not request, managing crawl access and server load. Critically, it controls crawling, not indexing — a blocked page can still be indexed if linked elsewhere. To keep a page out of search, use a noindex tag, not robots.txt, which is the single most misunderstood point.

Robots.txt sits at your domain root and issues instructions to crawlers before they fetch pages. It is ideal for steering bots away from low-value or resource-heavy areas — internal search, admin paths, faceted parameters — to protect crawl budget and server load. But it only governs whether a bot requests a URL.

The crucial distinction: blocking a URL in robots.txt does not remove it from the index. If other pages link to it, Google can still index the URL without crawling it, showing it with no description. To exclude a page from search, allow crawling and add a noindex tag, or use other removal tools. Confusing these two is the root of most robots.txt disasters, as Google’s robots.txt documentation stresses.

Correct Robots.txt Syntax

Robots.txt uses simple directives: User-agent specifies which crawler a rule applies to, Disallow blocks paths, and Allow creates exceptions. Rules are case-sensitive and matched by prefix. A Sitemap directive points crawlers to your XML sitemap. Small syntax errors have large consequences, so precision and testing are essential.

The structure is straightforward but unforgiving. Each block starts with User-agent naming the target crawler (or * for all), followed by Disallow and Allow rules that match URL paths by prefix. Disallow: / blocks the entire site; Disallow: /admin/ blocks that directory. Allow carves out exceptions within a disallowed path. Paths are case-sensitive, so /Admin and /admin differ.

Include a Sitemap directive pointing to your XML sitemap so crawlers discover it easily. Order and specificity matter for how conflicting rules resolve. Because a single stray slash can block far more than intended, always test changes in Search Console’s robots.txt tester before deploying, and treat the file as production code subject to review.

Costly Robots.txt Mistakes

The most damaging mistakes are blocking the whole site with Disallow: /, blocking CSS and JavaScript that Google needs to render pages, using robots.txt to hide pages that should be noindexed, and leaving staging blocks in production. Each can suppress rankings or deindex content, often without any obvious warning.

The catastrophic error is shipping a staging robots.txt — with Disallow: / blocking everything — to production, which tells Google to stop crawling the entire site. It happens more often than anyone admits, usually during launches. Equally harmful is blocking CSS and JavaScript resources: Google needs them to render pages, and blocking them can make your site look broken to the crawler, hurting rendering.

Using robots.txt to “hide” sensitive pages backfires twice — it does not prevent indexing, and the file is public, so you are advertising the paths you wanted hidden. Handle those with authentication and noindex instead. Audit robots.txt as part of every SEO audit and after every launch, since its failures are silent and severe.

Managing AI Crawlers

Robots.txt now also governs AI crawlers that gather content for training and answer engines. You can allow or block specific AI user-agents, but blocking them may reduce your visibility in AI answers. The decision is strategic: weigh control over how your content is used against presence in the growing AI search channel.

AI companies operate their own crawlers with distinct user-agent names, and robots.txt is the standard way to signal your preferences to them. You can disallow specific AI bots if you want to limit how your content is used for training or answer generation. Many publishers have made this choice deliberately.

But there is a trade-off. Blocking the crawlers that feed AI answer engines can remove you from a channel where being cited increasingly matters, as covered in AI search ranking factors. There is a difference between crawlers used purely for training and those that power live citations, so decide per-bot rather than blanket-blocking. Whatever you choose, be intentional and revisit it as the landscape shifts, tracking impact in your AI analytics and on your dashboard.

Frequently Asked Questions

The Bottom Line

Robots.txt is small, powerful, and dangerous in equal measure. Remember its one defining rule — it controls crawling, not indexing — write it precisely, never block rendering resources, and test every change before it ships. Increasingly it also governs your relationship with AI crawlers, a decision worth making deliberately. Treat this humble text file as production code, because a single wrong line can hide your entire site. Pair it with a clean XML sitemap for complete crawl control.