Afiliere Emag ProfitShare

[tetele]

Disallow in robots.txt doar opreste indexarea URL-urilor, nu limiteaza accesul crawlerului la ele N-o sa functioneze. Been there, done that.

[Nichita]

Disallow in robots.txt doar opreste indexarea URL-urilor, nu limiteaza accesul crawlerului la ele N-o sa functioneze. Been there, done that.

Robots blocheaza accesul botilor.


The robots.txt file is like an electronic No Trespassing sign. It tells Googlebot and other crawlers which files and directories on your server should not be crawled.

It's important to note that even if you use a robots.txt file to block spiders from crawling content on your site, Google could discover it in other ways and add it to our index.

https://support.google.com/webmaster...er/93708?hl=en