Robots.txt Checker

Read a site's robots.txt, see whether Googlebot, Bingbot and AI crawlers may crawl a URL, which rule decides, and what can't be determined.