Blocked by robots.txt: find the rule that blocks the page

Updated October 13, 2026 · Check Before Publish

“Blocked by robots.txt” means a Disallow rule in your site's robots.txt stops Googlebot from fetching the URL. To fix it, find the matching rule and narrow or remove it if the page should be crawled. robots.txt controls crawling, not indexing: to keep a page out of Google, allow crawling and use noindex.

Two statuses that look alike

• Blocked by robots.txt: Google isn't allowed to fetch the URL, so it is normally not indexed (there is a very small chance it still is, using information from other pages).
• Indexed, though blocked by robots.txt: Google indexed the URL without reading it, usually because other pages link to it. It can appear in results with no description, or a very limited one. If you want it out of Google, blocking it is the wrong tool: unblock it and add noindex. If you want it in Google, unblock it.

How a URL gets matched

A crawler reads the group of rules for its own name (for example User-agent: Googlebot) and falls back to User-agent: * only if there is no group for it. Within that group, the most specific (longest) matching rule wins, and for Google an Allow wins a tie. So in this file:

User-agent: *
Disallow: /events/
Allow: /events/open-studio

/events/open-studio may be crawled (the Allow rule is longer), while /events/archive may not. Common accidents:

• Disallow: / left over from a staging site, which blocks everything.
• A wildcard that matches more than intended, such as Disallow: /*? blocking every URL with a query string.
• A separate User-agent: Googlebot group added later. Googlebot then ignores the * group entirely, so rules written there stop applying to Google.

Find the rule that blocks your URL

Search Console's robots.txt report shows which robots.txt files Google found, when it last fetched them, and any warnings or errors. To see which rule blocks a URL, Google suggests using a robots.txt validator. Paste the page address below: the checker reads your live robots.txt, tells you whether Googlebot and Bingbot may crawl that URL, and shows the line that decides. It also reports what the file says to common AI crawlers such as GPTBot and ClaudeBot.

How to fix it

1. Decide whether the page should be crawled. Private, duplicate or utility URLs may be blocked on purpose; that's fine as long as you don't also need them kept out of the index (see below).
2. Edit the rule so it no longer matches: remove it, make it more specific, or add a longer Allow for the path you want crawled.
3. Edit the file that is actually served. robots.txt lives at the root of each host (https://www.example.com/robots.txt), and each subdomain has its own. Some platforms generate it for you; in that case change it through the platform's setting or template, because you may not be able to edit the file directly.

Don't combine robots.txt and noindex

If a page is blocked in robots.txt, Google can't read it, so it never sees a noindex tag on it. That is how pages end up “Indexed, though blocked”. To remove a page from Google: allow crawling, add noindex, and wait for it to drop out. See Excluded by ‘noindex’ tag for the noindex side.

How to confirm the fix

1. Run the check again: it should say Googlebot may crawl the URL, and name the rule that allows it (or that no rule applies).
2. In Search Console, the robots.txt report (under Settings, for domain-level properties only) shows the version Google last fetched. Google generally caches robots.txt for up to 24 hours, longer if it can't refresh it, so a change can take that long to be picked up. The report has a “Request a recrawl” option for urgent cases.
3. Use URL Inspection → Test live URL, then Request indexing, or Validate fix on the report.

Fixed this one? Before you send people to the page, run the full pre-publish check for everything else: indexing signals, link preview, broken links, alt text, structured data and HTTPS in one pass.