I’ve been running some tests on Googlebot to find out more about how it works. They are a work in progress, and the plan is to finish a massive article (or several) of my findings.
Unfortunately, Googlebot is not particularly interested in crawling the tests, as this is the only page that links to them. I’ll have to work on that.
I thought it would be interesting to share some of the tests so far.
- Image Tests – Which responsive images are loaded?
- Image Tests 2 – Images in Structured Data?
- Image Tests 3 – Images in a link?
- No Index – Does Googlebot render noindex pages?
- To No Index – Does Googlebot index JS based noindex pages?
- robots.txt – Does Googlebot honour it?
- Canonical tags – What if a page is canonicalized out?
- Missing Page – Does Googlebot render 404 pages?
- GoogleBot Navigator – How does Googlebot identify itself?
- GoogleBot Random – What do random numbers look like?
- GoogleBot JS Timeout – How long does Googlebot render for?
- GoogleBot Date.Now() – How long does it take to run code or timeouts?
- GoogleBot Date.Now() Excessive – Just to prove a point, and test timeouts.
- GoogleBot 5 Second Timeout – How long does it take in real time?
- GoogleBot 5 Second Request – How long does it take in Google time?
- GoogleBot Resources – How long do they take in Google time?
- GoogleBot Resources Chained – How long do they take in Google time?
- GoogleBot Resources Chained for 1 minute – Will Googlebot go on forever?
- GoogleBot Viewport – How tall can it be?
- GoogleBot Infinite Scroll – Does Googlebot see the dynamically added content?
- GoogleBot Caching – How much and how long?
- Large Page – Does Googlebot have a limit?
- GoogleBot Mixed – Randomised request loading with different types of delays.
- 0 Second Timeouts – Yielding.
- 301 Redirects – How long?
- 302 Redirects – Any different?
- JavaScript Redirects – Are they followed?
- noindex JavaScript Redirects – Does the noindex matter?
- Pinging when unloading the page – Is it reliable?
- Many redirects – How many will Googlebot follow?
The tests use several mechanisms:
- Logging of information to the head of the page so it can be seen in the URL Inspection HTML.
- Logging relevant browser events, including error, resize, visibilitychange, etc.
- Observing resource loading and logging their data.
- Wrapping setTimeout so can log when it starts and ends.
- Tracking real-time passing via server requests and their responses that add timestamps to the client log.
- Client-to-Server API requests that create GA4 events, including client/server times and bot, which are viewed in Looker Studio.
- Verification of the Googlebot IP address and identification of the documented user agents.
- Generation of performance measurements for visualising in the Chrome performance report.
- Resource requests have a cache buster, identify the page view they are part of and can specify how long before the server responds.
Can you think of any other Googlebot tests?