Blocked by robots.txt: find the rule that blocks the page

Updated · Check Before Publish

“Blocked by robots.txt” means a Disallow rule in your site's robots.txt stops Googlebot from fetching the URL. To fix it, find the rule that matches the URL and narrow or remove it if the page should be crawled. Remember what robots.txt is for: it controls crawling, not indexing. To keep a page out of Google, allow it to be crawled and use noindex instead.

Two statuses that look alike

  • Blocked by robots.txt: the URL is not indexed because Google isn't allowed to fetch it.
  • Indexed, though blocked by robots.txt: Google indexed the URL without reading it, usually because other pages link to it. It can appear in results with no description. If you want it out of Google, blocking it is the wrong tool: unblock it and add noindex. If you want it in Google, unblock it.

How a URL gets matched

A crawler reads the group of rules for its own name (for example User-agent: Googlebot) and falls back to User-agent: * only if there is no group for it. Within that group, the most specific (longest) matching rule wins, and for Google an Allow wins a tie. So in this file:

User-agent: *
Disallow: /events/
Allow: /events/open-studio

/events/open-studio may be crawled (the Allow rule is longer), while /events/archive may not. Common accidents:

  • Disallow: / left over from a staging site, which blocks everything.
  • A wildcard that matches more than intended, such as Disallow: /*? blocking every URL with a query string.
  • A separate User-agent: Googlebot group added later. Googlebot then ignores the * group entirely, so rules written there stop applying to Google.

Find the rule that blocks your URL

Google removed its robots.txt Tester from Search Console in December 2023; the robots.txt report that replaced it shows the files Google fetched and any parse problems, but not which rule applies to a given URL. Paste the page address below: the checker reads your live robots.txt, tells you whether Googlebot and Bingbot may crawl that URL, and shows the line that decides. It also reports what the file says to common AI crawlers such as GPTBot and ClaudeBot.

Opens the Robots.txt Checker and runs it. Free, no account. The address is not stored and never goes into the URL bar.

How to fix it

  1. Decide whether the page should be crawled. Private, duplicate or utility URLs may be blocked on purpose; that's fine as long as you don't also need them kept out of the index (see below).
  2. Edit the rule so it no longer matches: remove it, make it more specific, or add a longer Allow for the path you want crawled.
  3. Edit the file that is actually served. robots.txt lives at the root of each host (https://www.example.com/robots.txt), and each subdomain has its own. Some platforms generate it for you; in that case change it through the platform's setting or template, because an uploaded file may be ignored or overwritten.

Don't combine robots.txt and noindex

If a page is blocked in robots.txt, Google can't read it, so it never sees a noindex tag on it. That is how pages end up “Indexed, though blocked”. To remove a page from Google: allow crawling, add noindex, and wait for it to drop out. See Excluded by ‘noindex’ tag for the noindex side.

How to confirm the fix

  1. Run the check again: it should say Googlebot may crawl the URL, and name the rule that allows it (or that no rule applies).
  2. In Search Console, the robots.txt report (under Settings) shows the version Google last fetched. Google generally caches robots.txt for up to a day, so a change can take that long to be picked up.
  3. Use URL Inspection → Test live URL, then Request indexing, or Validate fix on the report.

Fixed this one? Before you send people to the page, run the full pre-publish check for everything else: indexing signals, link preview, broken links, alt text, structured data and HTTPS in one pass.

Related guides