Blocked by robots.txt (Submitted URL) in Search Console

By Amit Patel

“Submitted URL blocked by robots.txt” means you told Google to index a URL (via your sitemap) while your robots.txt file simultaneously tells Google not to crawl it — a direct contradiction Google resolves by obeying the block. The status is almost always a configuration mistake rather than a content problem, which makes it one of the more satisfying issues to fix: find the rule, correct it, and the pages usually recover on the next crawl.

What the Status Means

Your XML sitemap submitted the URL as a page you want indexed, but a Disallow rule in robots.txt prevents Googlebot from crawling it — so Google cannot fetch the content and will not index it.

The word “submitted” is the tell: it specifically flags URLs in your sitemap, which is why it reads as a contradiction rather than a deliberate choice. If you never wanted the URL indexed, it should not be in the sitemap in the first place. If you do want it indexed, the robots.txt rule is the bug. Understanding how the file works end to end is worth a read of the robots.txt guide before you touch anything.

Finding the Rule That Blocks It

Use the URL Inspection tool to confirm the block, then read your robots.txt top to bottom to find the specific Disallow pattern matching the path — remembering that the most specific matching rule wins in Google’s implementation.

The URL Inspection tool will state plainly that the URL is blocked and often names the crawl-blocking rule. Then open your robots.txt and look for a Disallow whose path prefix matches the URL — a broad rule like Disallow: /blog silently catches every blog URL, and wildcard rules (Disallow: /*?) catch every URL with a query string. Test candidate URLs against your rules before editing; a single over-broad line is the usual culprit.

Fixing It Without Breaking Other Pages

Narrow or remove the offending rule so the wanted URLs are crawlable, then re-test in URL Inspection and validate the fix — but change rules surgically, because robots.txt edits affect entire URL patterns at once.

The safest fix is to make the rule more specific rather than deleting a whole block: if Disallow: /products is catching real product pages, target only the parameter or path you actually meant to block. After editing, use the live test to confirm the URL is now allowed, resubmit the sitemap if needed, and click “Validate fix.” Keep an eye on the wider Page indexing report afterward to be sure you did not accidentally unblock URLs you wanted excluded — a robots.txt change ripples across every matching path. Google’s robots.txt documentation details the matching rules precisely.

Why Blocking Is Not How You Deindex

A common cause of this status is trying to remove a page from Google by blocking it in robots.txt — which backfires, because a blocked page can still be indexed without its content if other pages link to it.

To keep a page out of the index, let Google crawl it and serve a noindex directive; robots.txt blocking prevents the crawl that would let Google see the noindex in the first place. This is the distinction behind the related Excluded by ‘noindex’ tag status — noindex controls indexing, robots.txt controls crawling, and confusing the two produces exactly this error. If the URL genuinely should not be indexed, remove it from the sitemap and use noindex instead of a disallow.

Frequently Asked Questions

The Bottom Line

This status is a conflict you can resolve in minutes: your sitemap and your robots.txt disagree, and robots.txt is winning. Decide whether the URL truly belongs in the index, then align the two files — fix the crawl rule if you want the page, or drop it from the sitemap and noindex it if you do not. Keep robots.txt and indexing directives in their proper lanes and this error stops recurring, a discipline the rest of the Search Console cluster reinforces.