What does “Blocked due to access forbidden (403)” mean?
“Blocked due to access forbidden (403)” means Googlebot got a 403 from your server, so the page is not indexed. Google says Googlebot never sends credentials, so it calls that response incorrect if you want the page indexed. Look at what your server, CDN and firewall send to Googlebot, not what your browser receives.
The short version
- Google's wording: “HTTP 403 means that the user agent provided credentials, but was not granted access. However, Googlebot never provides credentials, so your server is returning this error incorrectly. The page will not be indexed.”
- Google's remedy: let signed-out visitors in, or explicitly allow Googlebot requests without authentication, after verifying its identity. For the 401 sibling status, “Blocked due to unauthorized request (401)”, Google says the same.
- Google treats all 4xx errors except 429 alike: it tells the next system that the content doesn't exist, and URLs already indexed that return a 4xx status are removed from the index. It gives no timeline.
- Do not use 401 or 403 to slow Googlebot down. Google says 4xx codes other than 429 “have no effect on crawl rate”.
- Security tools answer blocked requests with 403. Cloudflare says the HTTP status code of a block is “403 (most security features)”, and AWS WAF's Block action responds “by default” with 403.
Who is allowed to claim to be Googlebot
Google warns that the user agent string “can be spoofed”, so a firewall cannot trust the name alone. Google's way to verify a request is a reverse DNS lookup on the IP address, then a forward lookup on the result, or matching the IP against its published ranges. The domain must be googlebot.com, google.com or googleusercontent.com, and the forward lookup must return the original IP address. Google's own example:
host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1Cloudflare and AWS both build on this. Cloudflare keeps a list of verified bots and has managed rules that flag a request whose user agent says Googlebot but whose source cannot be verified. AWS WAF Bot Control adds a label to verified bots and says its rule group does not block them. Both let you write your own rules that exempt verified bots.
Find what is refusing Googlebot
- In Search Console, open URL Inspection for the URL and run Test live URL. Google says the live test “does not test for all possible indexing issues”, and that server errors can be transient between a live test and a crawl.
- Look in your server, CDN and firewall logs for requests from addresses that verify as Google, and read the status they got. That is the most direct evidence of what Googlebot received.
- Check bot-protection, rate-limit, country and path rules. If a rule challenges or blocks by user agent, make sure real, verified Googlebot requests are exempt (Cloudflare documents an exception for its fake-bot rules; AWS documents rules on the verified-bot label).
- If the page is meant to be behind a login, a 401 or 403 is the correct answer and the status is working as intended. Decide whether it should be public.
See what a visitor like us gets
Paste the URL. The checker reports the status the page answers to our server, and says plainly when the site answered 403 or the answer was not an HTML page.
What we tested
On 2026-10-13 we ran the live Website Pre-Publish Checker on three addresses, and a plain request test on eight real sites.
| What we entered | What the checker reported |
|---|---|
https://www.nytimes.com/ | HTTP 403: “The site answered 403 to our checker”, with the advice to open the page in a browser; if it loads normally, only our checker was refused. |
| A public test service that always answers 403 (plain text) | “Couldn't check this URL. This URL returned text/plain; charset=utf-8 (status 403), not an HTML page, so it can't be checked.” |
| The same service answering 401 | The same message with status 401. |
For the second test we requested each page's address with curl from an ordinary machine, three times, changing only the User-Agent header: a Chrome browser string, the Googlebot string, and curl's default. The source IP was never Google's.
| Site | Browser string | Googlebot string | curl default |
|---|---|---|---|
| nytimes.com | 403 | 403 | 403 |
| stackoverflow.com | 403 | 403 | 403 |
| reuters.com | 401 | 401 | 401 |
| en.wikipedia.org | 200 | 403 | 200 |
| bbc.com, github.com, cloudflare.com, theguardian.com | 200 | 200 | 200 |
What we can say: four of the eight sites sent a 401 or 403 to at least one of these requests, and Wikipedia answered 200 to two identities and 403 to the one that claimed to be Googlebot from a non-Google address. That is consistent with sites checking a claimed Googlebot, but we did not investigate why, and we cannot say what any of them sends to the real Googlebot. A 403 to a script also does not mean a person with a browser sees one.
What this can't tell you
We fetch from our own server. A 403 to us does not prove Googlebot got a 403, and a 200 to us does not prove it got a 200: only URL Inspection and your own logs show that. We also cannot see your firewall rules or tell you which one fired.
Related: why a page isn't showing up on Google and Blocked by robots.txt (a different block: the crawler is told not to ask).
Sources
We read these official pages on 2026-10-13.
- Page indexing report - Search Console Help
- How HTTP Status Codes Affect Google's Crawlers (page dated 2026-02-04)
- Verify Requests from Google Crawlers and Fetchers (page dated 2026-03-20)
- Google's common crawlers (page dated 2026-07-14)
- URL Inspection tool - Search Console Help
- Actions reference - Cloudflare Ruleset Engine docs
- Verified bots - Cloudflare bot solutions docs
- Fake bot detection blocking legitimate requests - Cloudflare WAF docs
- Using rule actions in AWS WAF
- AWS WAF Bot Control rule group
- 403 Forbidden - MDN
- 401 Unauthorized - MDN
- RFC 9110: HTTP Semantics