Despre Robots.txt

[gemanaru]

Omule ma intereseaza in mod deosebit acest fisier robots.txt .Pentru ca VREAU sa ma indexeze si pe mine google

Da, trebuie sa recunosc ca am un site care vreau sa mi-l indexeze .

Adresa este : tomistravel.ro

[w!ll]

Pentru inceput ar trebui sa inlocuiesti acel redirect din metataguri, cu redirect 301 si lasa robots.txt in pace

[sovidiu]

Pune totusi ceva cu inteles in robots.txt. Momentan, acolo ai:

------------
User-agent: *

Sitemap: http://www.numele_site-ului_tau.ro/sitemap.xml
------------

User-agent ce?

Pe randul de sub "User-agent: *" poti pune:

# indexeaza tot
Allow: /

sau

# indexeaza tot
Allow: /

# nu indexa folder-ul "personal"
Disallow: /personal/

# nu indexa fisierul "personal.html"
Disallow: /personal.html

# nu indexa foldere sau fisiere ce incep cu structura "personal"
Disallow: /personal


In Google Webmaster Tools ai pe undeva pe-acolo un generator de fisier robots.txt si o sectiune unde sa verifici daca ai un robots.txt valid.

Completeaza si formularul de pe Alexa.com si adauga datele si intr-un fisier info.txt accesibil la numele_site-ului_tau.ro/info.txt

[d_alinus2004]

foarte bun topicu asta despre robots.txt in sfarsit m'am luminat si eu..

[Dude]

In cat timp pot vedea rezultatele (paginile scoase din goolge) dupa ce pun regulile in robots.txt ? Am niste pagini "interzise" mentionate in robots.txt care inca apar in rezultate la 2 zile dupa editarea fisierului.

[BladHaund]

Ca sa mearga mai repede, poti sa o faci pentru Google si din webmaster tools: Tools->Remove URLs.

Dar:

We try to complete this process in 3-5 business days, though large-scale URL removal requests will take longer.

[Dude]

Ok. Atunci astept in continuare rezultatele.

[eNoston]

Sper sa va fie de folos (se adauga la robots.txt)

###
#Unsafe robots to keep away
###
User-agent: QihooBot
Disallow: /

User-agent: OmniExplorer_Bot
Disallow: /

User-agent: TMCrawler
Disallow: /

User-agent: exabot
Disallow: /

User-agent: tm.net.my
Disallow: /

User-agent: Twiceler
Disallow: /

User-agent: mj12bot
Disallow: /

User-agent: asterias
Disallow: /

User-agent: Jyxobot/1
Disallow: /

User-agent: BackDoorBot/1.0
Disallow: /

User-agent: Black Hole
Disallow: /

User-agent: BotALot
Disallow: /

User-agent: BuiltBotTough
Disallow: /

User-agent: Bullseye/1.0
Disallow: /

User-agent: CopyRightCheck
Disallow: /

User-agent: DittoSpyder
Disallow: /

User-agent: EmailCollector
Disallow: /

User-agent: EmailSiphon
Disallow: /

User-agent: EmailWolf
Disallow: /

User-agent: EroCrawler
Disallow: /

User-agent: ExtractorPro
Disallow: /

User-agent: FairAd Client
Disallow: /

User-agent: Flaming AttackBot
Disallow: /

User-agent: Foobot
Disallow: /

User-agent: Gaisbot
Disallow: /

User-agent: grub
Disallow: /

User-agent: grub-client
Disallow: /

User-agent: Harvest/1.5
Disallow: /

User-agent: hloader
Disallow: /

User-agent: InfoNaviRobot
Disallow: /

User-agent: Iron33/1.0.2
Disallow: /

User-agent: JennyBot
Disallow: /

User-agent: larbin
Disallow: /

User-agent: LexiBot
Disallow: /

User-agent: LinkextractorPro
Disallow: /

User-agent: LinkScan/8.1a Unix
Disallow: /

User-agent: LinkWalker
Disallow: /

User-agent: LNSpiderguy
Disallow: /

User-agent: lwp-trivial
Disallow: /

User-agent: lwp-trivial/1.34
Disallow: /

User-agent: Mata Hari
Disallow: /

User-agent: Mister PiX
Disallow: /

User-agent: Offline Explorer
Disallow: /

User-agent: Openbot
Disallow: /

User-agent: Openfind
Disallow: /

User-agent: Openfind data gathere
Disallow: /

User-agent: Oracle Ultra Search
Disallow: /

User-agent: SiteSnagger
Disallow: /

User-agent: SpankBot
Disallow: /

User-agent: spanner
Disallow: /

User-agent: Titan
Disallow: /

User-agent: True_Robot
Disallow: /

User-agent: True_Robot/1.0
Disallow: /

User-agent: turingos
Disallow: /

User-agent: URL Control
Disallow: /

User-agent: URL_Spider_Pro
Disallow: /

User-agent: URLy Warning
Disallow: /

User-agent: VCI
Disallow: /

User-agent: VCI WebViewer VCI WebViewer Win32
Disallow: /

User-agent: Web Image Collector
Disallow: /

User-agent: WebAuto
Disallow: /

User-agent: WebBandit
Disallow: /

User-agent: WebBandit/3.50
Disallow: /

User-agent: WebCopier
Disallow: /

User-agent: WebEnhancer
Disallow: /

[eNoston]

eu folosesc auditmypc.com ca sa fac sitemap-ul site-ului. cu ajutorul lui vad si de ex. daca folderul themes sau alt folder ex images e indexabil,

daca nu le vreau indexabile le adaug la robots.txt

Disallow: /sectiuni care nu se doresc indexate/

[CODEX]

pentru un forum gen vbulletin trebuie facut ceva altfel sau se face ca la orice site normal ?