Robots blocheaza accesul botilor.Postat în original de tetele
Disallow in robots.txt doar opreste indexarea URL-urilor, nu limiteaza accesul crawlerului la eleN-o sa functioneze. Been there, done that.
The robots.txt file is like an electronic No Trespassing sign. It tells Googlebot and other crawlers which files and directories on your server should not be crawled.
It's important to note that even if you use a robots.txt file to block spiders from crawling content on your site, Google could discover it in other ways and add it to our index.
https://support.google.com/webmaster...er/93708?hl=en
Răspunde cu citat