Uite aicea raspunsul la intrebarea ta:

The AdSense crawler is different from the Google crawler
The two crawlers are separate, but they do share a cache. We do this to avoid both crawlers requesting the same pages, thereby helping publishers conserve their bandwidth. Similarly, the Sitemaps crawler is separate.
LINK

Raport Robots.txt - acces
Alte informatii...


Cum blochezi google boot..