Robots.txt Checker
May search engines crawl this URL? Read the site's robots.txt, see which rule applies to Googlebot, Bingbot and AI crawlers, and what can't be determined.
What it checks
- Whether robots.txt loads, and what that means: 200 = its rules apply; 404 = nothing is blocked; server errors = Google pauses crawling the site.
- For the URL you enter: whether Googlebot and Bingbot may crawl it, and the exact rule and line that decides (longest match wins, Allow wins a tie,
*and$supported). A block is Fix before publishing; if robots.txt can't be read, the answer is Not checked, never Passed. - AI and dataset crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot, Applebot-Extended): reported as allowed or blocked — that's your choice, not an error.
- Syntax: rules before any User-agent, misspellings, Noindex lines (unsupported), lines crawlers ignore, relative Sitemap URLs, files over 500 KiB, an HTML page served as robots.txt.
Example
A staging robots.txt with User-agent: * and Disallow: / went live with the site. Enter the new page's URL: the check shows “Googlebot is blocked from this URL — Disallow: / (line 2)”.
FAQ
Does blocking in robots.txt remove a page from search?
No. It stops crawling. A blocked URL can still be listed if other pages link to it; to keep a page out of search, allow crawling and use noindex.
Which rules do you follow?
The robots.txt standard (RFC 9309), plus Google's documented behaviour where the standard leaves room, such as accepting common misspellings.
Why is it Not checked when robots.txt times out?
Because we can't know what crawlers receive. Google treats an unreachable robots.txt as “don't crawl”, so check it in a browser.