Rezultate 1 la 4 din 4

Subiect: Yahoo ignora robots.txt

  1. #1
    Avatarul lui Razvan Pop
    Razvan Pop este deconectat Membru SeoPedia
    Reputatie:
    41
    Data înscrierii
    9th July 2004
    Locaţie
    Bucuresti
    Vârstă
    44
    Posturi
    1.376
    Putere Rep
    41


    Implicit Yahoo ignora robots.txt

    Salut,

    A mai observat careva daca Yahoo Slurp ignora robots.txt? Pe mine ma enerveaza la culme, indexeaza toate aiurelile de pe forum.
    Cu stima,
    Razvan Pop

  2. #2
    Avatarul lui Cristian Mezei
    Cristian Mezei este deconectat Administrator
    Reputatie:
    63
    Data înscrierii
    20th February 2005
    Locaţie
    Everywhere
    Posturi
    6.975
    Putere Rep
    63


    Implicit

    http://www.robotstxt.org/wc/active/html/index.html

    Eu din cate stiu Slurp, nu adera la regulile si lista oficiala de roboti.

    Am mai auzit spuse, cum ca ar ignora anumite tag-uri standard din robots.txt
    Co-founder of Deepstash. Tech lover. Dinosaur. Dad. Chef.

  3. #3
    Avatarul lui Razvan Pop
    Razvan Pop este deconectat Membru SeoPedia
    Reputatie:
    41
    Data înscrierii
    9th July 2004
    Locaţie
    Bucuresti
    Vârstă
    44
    Posturi
    1.376
    Putere Rep
    41


    Implicit

    Cu stima,
    Razvan Pop

  4. #4
    Avatarul lui Cristian Mezei
    Cristian Mezei este deconectat Administrator
    Reputatie:
    63
    Data înscrierii
    20th February 2005
    Locaţie
    Everywhere
    Posturi
    6.975
    Putere Rep
    63


    Implicit

    Stiu ce spun ei ...

    Se pare ca spun degeaba.

    Sau poate cei care trec in robots.txt .. nu pun user-agent "Slurp"

    Dupa cum o sa vezi mai jos... SUnt mai multe variante de spider Yahoo

    Slurp.so/1.0= Yahoo.com
    Slurp/2.0j=Yahoo.com
    Slurp/2.0-KiteHourly=Yahoo.com
    Slurp/2.0-OwlWeekly=Yahoo.com
    Slurp/3.0-AU=Yahoo.com
    slurp@inktomi.com=Yahoo.com
    slurp@inktomi= Yahoo.com
    Slurp=Yahoo.com
    Iata o lista cu toti robotii , si user-agentul corespunzator.

    Daca nu e bine pus aici razvan ... Pune postul altundeva

    :robot/1.0=Neofonie.de
    0.1_hseo(at)cs.rutgers.edu=Rutgers.edu
    AbachoBOT (Mozilla compatible)=Crawler.de
    AbachoBOT=Crawler.de
    abcdatos_botlink=Abcdatos.com
    About/0.1libwww-perl/5.47=About.com
    AcoiRobot=Acoi Picture Robot
    Acoon Robot v1.50.001=Acoon.de
    Acoon Robot v1.52=Acoon.de
    Acoon Robot=Accon.de
    ActiveWorlds/3.xx=ActiveWorlds.com
    admin@crawler.de=Crawler.de
    admin@maxbot.com=maxbot.com
    ADSARobot=ADSARobot
    AESOP_com_SpiderMan=Aesop.com
    agadine/1.x.x=Agada.de
    Agent-admin/=Daum.net
    ah-ha.com crawler (crawler@ah-ha.com)=Ah-ha.com
    ah-ha.com crawler=Ah-ha.com
    Aladin/3.324=Aladin.de
    Aleksika Spider/1.0=Aleksika.de
    AlkalineBOT/1.3=Vestris.com
    AlkalineBOT/1.4=Vestris.com
    AlkalineBOT=AlkalineBOT
    Allesklar/0.1 libwww-perl/5.46=Allesklar.de
    AltaVista-Intranet=Altavista.co.uk
    AmfibiBOT=Amfibi.com
    Amiga-AWeb/3.4.167SE=AWeb.com
    AnswerChase PROve x.0=AnswerChase.com
    AnswerChase x.0=AnwserChase.com
    AnswerChase=AnswerChase.com
    Antibot=AntiSeach.net
    antibot-V1.1.5/i586-linux-2.2=AntiSeach.net
    AnzwersCrawl/2.0=Anzwers
    A-Online Search=Jet2Web.com
    Aport=Aport.ru
    appie 1.1=Walhello.com
    appie=Walhello.com
    arachnoidea@euroseek.net=Euroseek.net
    Arachnoidea=Euroseek.com
    Aranha=Girafa.com
    ArchitectSpider=Excite.com
    archive_org=Archive.org
    archive.org_bot=Archive.org
    ask jeeves=Ask Jeeves
    ask.24x.info=Ask.24x.info
    ASPSeek/1.2.5=ASPSeek.org
    ASPseek/1.2.9d=ASPSeek.com
    ASPSeek/1.2.x=ASPSeek.org
    ASPSeek/1.2.xa=ASPSeek.org
    ASPseek/1.2.xx=ASPseek.org
    ASPSeek/1.2.xxpre=ASPSeek.org
    ASPSeek=ASPSeek.org
    asterias/2.0=SingingFish.com
    Asterias=SingingFish.com
    Atomz/1.0=Atomz.com
    Atomz=Atomz.com
    Augurfind=Augurnet.ch
    augurnfind V-1.x=Augurnet.ch
    autohttp=Elsop.com
    AV Fetch 1.0=Altavista.com
    AVSearch=Altavista.com
    AVSearch-3.0=Altavista.com
    Baiduspider+=Baidu.com
    BaiDuSpider=Baidu.com
    BanBots/1.2=Banbots.com
    BarraHomeCrawler=BarraHome.com
    bdcindexer_2.6.2=Business.com
    BDFetch=Brandimensions.com
    BDNcentral Crawler v2.3=Bdncentral.com
    beautybot/1.0=Cosmoty.com
    Bilbo/2.3b-UNIX=Biblo Bot
    BlackWidow=BlackWidow Bot
    Blaiz-Bee/1.0 (+http://www.blaiz.net)=Blaiz.net
    Blaiz-Bee/1.0=Blaiz.net
    BlitzBOT@tricus.net (Mozilla compatible)=Blitzsuche.rp-online.de
    BlitzBOT@tricus.net=Blitzsuche.rp-online.de
    BlogBot/1.x=Blogdex.net
    BlogBot=Blogdex.net
    blogWatcher_Spider/0.1=Blogwatcher.pi.titech.ac.jp
    boitho.com-dc/0.xx=Boitho.com
    boitho.com-robot/1.x=Boitho.com
    BSDSeek/1.0=Inktomi.com
    BullsEye=Intelliseek.com
    bumblebee@relevare.com=Relevare.com
    Buscaplus Robi/1.0=Buscaplus.com
    Buscaplus Robi=Buscaplus.com
    CanSeek/=Cansee.ca
    ChristCRAWLER 2.0=Christcrawler.com
    ChristCRAWLER=Christcrawler.com
    CipinetBot=Cipinet.com
    citenikbot/=Cenik.co.uk
    ClariaBot/1.0=Searchscout.com
    Claymont.com=Claymont.com
    Clushbot/3.3-BinaryFury=Clush.com
    Clushbot/3.3-BinaryFury (+http://www.clush.com/bot.html)=Clush.com
    CLIPS-index=Cips-index.imag.fr
    Computer_and_Automation_Research_Institute_Crawler =Research Bot
    contact/jylee@kies.co.kr=Kies.co.kr
    Convera Internet Spider V6.x=Convera.com
    cosmos/0.8_(robot@xyleme.com)=Xyleme.com
    cosmos/0.9_(robot@xyleme.com)=Xyleme.com
    Combine/x.0=Combine Bot
    cosmos=Xyleme.com
    C-PBWF-ip3000.com-crawler=IP3000.com
    CrawlConvera0.1=Converna.net
    Crawler admin@crawler.de=Crawler.de
    Crawler V 0.2.x admin@crawler.de=Crawler.de
    crawler@alexa.com=Alexa.com
    crawler@brainbot.com=Brainbot.com
    crawler@fast.no=Alltheweb.com
    Crawler=Crawler.de
    CrawlerBoy Pinpoint.com=Pinpoint.com
    CrawlerBoy=Pinpoint.com
    CrocCrawler= CrocCrawler.com
    Custom Spider www.bisnisseek.com /1.0=Bisnisseek.com
    DaAdLe.com ROBOT/=Daadle.com
    DataparkSearch/4.xx=DataparkSearch.com
    DaviesBot/1.7=Wholeweb.net
    DaviesBot=Wholeweb.net
    dbDig=Prairielandconsulting.com
    DBrowse 1.4b=DBrowse Bot
    DBrowse 1.4d=DBrowse Bot
    dCSbot/1.1=Openmarket.com
    DeepIndex=DeepIndex.com
    DeepIndexer.ca=DeepIndex.com
    deepweb=Deepweb.com
    Demo Bot DOT 16b= Btopenworld.com
    DiaGem/=Skyrocket.gr.jp
    DiaGem/1.1 (http://www.skyrocket.gr.jp/diagem.html)=Skyrocket.gr.jp
    DiaGem/1.1=Skyrocket.gr.jp
    Diamond/1.0=Searchscout.com
    Digger/1.0 JDK/1.3.0rc3=Diggit.com
    Digimarc WebReader=Digimarc.com
    DigOut4U=Arisem.com
    DIIbot/1.2=Digital-Integrity.com
    DIIbot=Digital-Integrity.com
    DittoSpyder=Ditto.com
    dloader(NaverRobot)/=Naver.com
    dtSearchSpider=Dtsearch.com
    Dumbot(version 0.1 beta - dumbfind.com)=Dumbfind.com
    Dumbot(version 0.1 beta)=Dumbfind.com
    dumrobo(NaverRobot)/=Naver.com
    EARTHCOM.info/1.4beta=Earthcom.info
    EasyDL/3.02=Keywen.com
    EasyDL/3.03=Keywen.com
    EasyDL/3.04=Keywen.com
    EasyDL/3.xx http://keywen.com/Encyclopedia/Bot=Keywen.com
    EasyDL/3.xx=Keywen.com
    EchO!/2.0=Voila.com
    egothor/3.0a (+http://www.xdefine.org/robot.html)=Xdefine.com
    egothor/3.0a=Xdefine.com
    EgotoBot/4.8=Egoto.com
    Enterprise_Search/1.0.xxx=Innerprise.net
    Enterprise_Search/1.0=Innerprise.net
    InnerpriseBot/1.0=Innerprise.net
    InnerpriseBot/1.0 (http://www.innerprise.com/)=Innerprise.net
    erik@malfunction.org=Malfunction.org
    EroCrawler=Erocrawler.com
    ES.NET_Crawler/2.0=Innerprise.net
    eseek-larbin_2.6.2=Exactseek.com
    ESISmartSpider=Travel-Finder.com
    ExactSeek Crawler/0.1=Exactseek.com
    exactseek-crawler-2.63=Exactseek.com
    Exalead NG/MimeLive Client=Exabot.com
    Excalibur Internet Spider V6.5.4=Excalib.com
    ExperimentalHenrytheMiragoRobot=Mirago.co.uk
    EyeCatcher (Download-tipp.de)/1.0=Download-tipp.de
    EyeCatcher=Download-tipp.de
    EZResult=Ezresults.com
    Fast Crawler Gold Edition=Alltheweb.com
    FAST Data Search Crawler=Alltheweb.com
    FAST Enterprise Crawler 6 (Experimental)=Alltheweb.com
    FAST Enterprise Crawler 6 used by FAST (FAST)=Alltheweb.com
    FAST Enterprise Crawler 6 used by FAST=Alltheweb.com
    FAST Enterprise Crawler 6=Alltheweb.com
    FAST Enterprise Crawler/6 (www.fastsearch.com)=Alltheweb.com
    FAST Enterprise Crawler/6=Alltheweb.com
    FAST FirstPage retriever (compatible; MSIE 5.5; Mozilla/4.0)=Alltheweb.com
    FAST FirstPage retriever=Alltheweb.com
    Fast PartnerSite Crawler=Alltheweb.com
    FastCrawler 3.0.1 (crawler@1klik.dk)=Alltheweb.com
    FastCrawler 3.0.1=Alltheweb.com
    FAST-WebCrawler/2.2.10=Altheweb.com
    FAST-WebCrawler/2.2.6=Altheweb.com
    FAST-WebCrawler/2.2.7=Altheweb.com
    FAST-WebCrawler/2.2.8=Altheweb.com
    FAST-WebCrawler/3.2 test=Altheweb.com
    FAST-WebCrawler/3.2=Altheweb.com
    FAST-WebCrawler/3.3=Altheweb.com
    FAST-WebCrawler/3.4/Nirvana=Altheweb.com
    FAST-WebCrawler/3.4/PartnerSite=Altheweb.com
    FAST-WebCrawler/3.5=Altheweb.com
    FAST-WebCrawler/3.6/FirstPage=Altheweb.com
    FAST-WebCrawler/3.6=Altheweb.com
    FAST-WebCrawler/3.7/FirstPage=Altheweb.com
    FAST-WebCrawler/3.7=Altheweb.com
    FAST-WebCrawler/3.8/Fresh=Altheweb.com
    FAST-WebCrawler/3.8=Altheweb.com
    FAST-WebCrawler/3.x Multimedia (mm dash crawler at fast dot no)=Altheweb.com
    FAST-WebCrawler/3.x Multimedia=Altheweb.com
    FAST-WebCrawler=Alltheweb.com
    Faxobot/1.0=FaXo.com
    Felix - Mixcat Crawler=Mixcat.com
    Felix=Mixcat.com
    FFC Trap Door Spider=Frequentfinders.com
    fido/1.0 Harvest/1.4.pl2=Planet Search
    FinaleRobot=expressus.com
    FindAnISP.com_ISP_Finder_v99a=FindAnISP.com
    FineBot=Finesearch.com
    Firefly/1.0 (compatible; Mozilla 4.0; MSIE 5.5)=Fireball.de
    Firefly/1.0=Fireball.de
    FlickBot 2.0 RPT-HTTPClient/0.3-3=DivX.com
    Fluffy the spider=SearchHippo.com
    Flunky=Metacarta.com
    Francis/1.0 (francis@neomo.de http://www.neomo.de/)=Neomo.de
    Francis/1.0=Neomo.de
    FreeFind.com-SiteSearchEngine/1.0=Freefind.com
    FyberSearch=Fybersearch.com
    FyberSpider=Fybersearch.com
    Gagglebot=Gagglebot
    GAIS Robot/1.0B2=Seed.net.tw
    Gaisbot/3.0+=Seed.net.tw
    Gaisbot/3.0=Seed.net.tw
    GalaxyBot/1.0 (http://www.galaxy.com/galaxybot.html)=Galaxy.com
    GalaxyBot/1.0=Galaxy.com
    GalaxyBot=Galaxy.com
    GammaSpider/1.0=Gammasite.com
    gazz/1.0=Infobee.ne.jp
    gazz@nttrd.com=Infobee.ne.jp
    Geckobot=Geckobot.com
    geckobot=Geckobot.com
    GenCrawler=Gendoor.com
    generic_crawler/01.0217/=Cmu.edu
    GentleSpider=Research.att.com
    GeonaBot=Geona.com
    GigaBaz=Brainbot.com
    GigaBazVStheWeb=Brainbot.com
    GigaBlast=Gigablast.com
    Gigabot/1.0=Gigablast.com
    GigabotSiteSearch/2.0 (sitesearch.gigablast.com)=Gigablast.com
    GigabotSiteSearch/2.0=Gigablast.com
    Gigabot=Gigablast.com
    Giskard=oralco.com
    GNODSPIDER=Gnod.net
    Goblin/0.9 (http://www.goguides.org/)=Goguides.org
    Goblin/0.9.x (http://www.goguides.org/goblin-info.html)=Goguides.org
    Goblin/0.9.x=Goguides.org
    Goblin/0.9=Goguides.org
    GoForIt.com=Goforit.com
    Goofer/0.2=Wanadoo.fr
    Google=Google.com
    Googlebot/1.0=Google.com
    Googlebot/2.1= Google.com
    Googlebot/Test=Google.com
    googlebot@googlebot.com=Google.com
    Googlebot=Google.com
    Googlebot-Image/1.0=Google.com Image Bot
    GrigorBot 0.8=Grigor.biz
    grub-client=Grub.org
    grub crawler(http://www.grub.org)=Grub.com
    gsa-crawler (Enterprise; GID-01422; jplastiras@google.com)=Google.com
    gsa-crawler (Enterprise; GID-01742;gsatesting@rediffmail.com)=Google.com
    gsa-crawler=Google.com
    Gulliver/1.2=Northernlight.com
    Gulliver/1.3=Northernlight.com
    Gulliver=Northernlight.com
    Gulper Web Bot 0.2.4=Gulper Bot
    Gulper Web Bot=Gulper Bot
    Hatena Antenna/0.4 (http://a.hatena.ne.jp/help#robot)=a.hatena.ne.jp
    Hatena Antenna/0.4=a.hatena.ne.jp
    HeinrichderMiragoRobot=Mirago.de
    HenrytheMiragoRobot=Mirago.co.uk
    HenryTheMiragoRobot=Mirago.co.uk
    Hippias/0.9 Beta= Hippias Bot
    holmes/x.x=Morfeo.centrum.cz
    HomePageSearch(hpsearch.uni-trier.de)=Hpsearch.uni-trier.de
    Homerbot: www.homerweb.com=Homerweb.com
    Homerbot=Homerweb.com
    htdig/3.1.6 (http://computerorgs.com)=Computerorgs.com
    htdig/3.1.x (root@localhost)=Htdig.org
    htdig/3.1.x=Htdig.org
    http://www.abcdatos.com/botlink/=Abcdatos.com
    Hubater=Hubat.com
    ia_archiver/1.6=Archive.org
    ia_archiver=Archive.org
    IAArchiver-1.0=Archive.org
    icsbot-0.1=Icseoul.org
    ideare - SignSite/1.x=Ideare.com
    Ideare - SignSite=Ideare.com
    IlTrovatore-Setaccio (+http://www.iltrovatore.it)=Iltrovatore.it
    Iltrovatore-Setaccio/0.3-dev=Iltrovatore.it
    IlTrovatore-Setaccio/1.2=Iltrovatore.it
    iltrovatore-setaccio/1.2-dev=Iltrovatore.it
    IlTrovatore-Setaccio=Iltrovatore.it
    IncyWincy=Incywincy.com
    IndexTheWeb.com Crawler7=IndexTheWeb.com
    Inet library=Inetlibrary.com
    info@searchhippo.com=Searchhippo.com
    InfoFly/1.0 (http://www.versions-project.org/)=Versions-project.org
    InfoFly/1.0= Versions-project.org
    InfoSeek Sidewinder/0.9=Infoseek.com
    InfoSeek Sidewinder/1.0A= Infoseek.com
    InfoSeek Sidewinder/1.1A= Infoseek.com
    Infoseek SideWinder/1.45 (Compatible; MSIE 10.0; UNIX)= Infoseek.com
    Infoseek SideWinder/1.45= Infoseek.com
    Infoseek SideWinder/2.0B (Linux 2.4 i686)= Infoseek.com
    Infoseek SideWinder/2.0B= Infoseek.com
    InfoSeek Sidewinder=infoseek.com
    Inktomi Search=Yahoo.com
    Inktomi=Yahoo.com
    InsumaScout=InsumaScout
    Internet Ninja 6.0=Dti.ne.jp
    InternetSeer.com= InternetSeer.com
    InternetSeer=InternetSeer
    InternetSeer=InternetSeer.com
    ip3000.com=ip3000.com
    ip3000.com-crawler=ip3000.com
    IPiumBot laurion(dot)com=Laurion.com
    ipiumBot=Laurion.com
    Iron33=Verno.ueda.info.waseda.ac.jp
    Jabot/6.x (http://odin.ingrid.org/)=Ingrid.org
    Jabot/6.x=Ingrid.org
    Jabot/7.x.x (http://odin.ingrid.org/)=Ingrid.org
    Jabot/7.x.x=Ingrid.org
    Jack=Domanova.co.uk
    jan.gelin@av.com=Altavista.co.uk
    jeeves=Ask Jeeves
    Jetbot/1.0=Jeteye.com
    JoocerBot=Joocer.com
    Jyxobot/x=Jyxo.cz
    Katriona=Katriona
    KE_1.0/2.0 libwww/5.2.8=Voila.fr
    Kenjin Spider=Kenjin.ne.jp
    KIT_Fireball/2.0=Fireball.de
    KIT-Fireball/2.0 (compatible; Mozilla 4.0; MSIE 5.5)=Fireball.de
    KIT-Fireball/2.0=Fireball.de
    KIT-Fireball=Fireball.de
    Knowledge.com/0.x=Knowledge.com
    Kototoi/0.1=s.u-tokyo.ac.jp
    kuloko-bot/0.2=Kuloko.com
    kuloko-bot/0.x=Kuloko.com
    Kulokobot=Kuloko.com
    kulturarw3/0.1=National Library of Sweden
    LEIA/3.01pr (LEIAcrawler; [SNIP])=Gseek.com
    LEIA=Gseek.com
    LexiBot=Lexibot.com
    LNSpiderguy=Lexis-nexis.com
    Look.com=Look.com
    lwp-trivial/1.27=Search4free.com
    lwp-trivial/1.34=Search4free.com
    Lycos_Spider_(T-Rex)/3.0=Lycos.com
    Lycos_Spider_(T-Rex)=Lycos.com
    lycos=Lycos.com
    Mackster( http://www.ukwizz.com )=Ukwizz.com
    Mackster=Ukwizz.com
    Mag-Net=Wind.it
    mailto:webcraft@bea.com=Beasys.com
    mammoth/1.0 (+http://www.sli-systems.com/)=Sli-Systems.com
    mammoth/1.0=Sli-Systems.com
    MantraAgent=Looksmart.com
    mapper@teradex.com=Mapper.teradex.com
    Mariner/5.1b [de] (Win95; I ;Kolibri gncwebbot)=Kolibri.de
    Mariner/5.1b= Kolibri.de
    Martini=Looksmart.com
    Marvin v0.3=Marvin Bot
    marvin/infoseek=Webseek.de
    marvin-team@webseek.de=Webseek.de
    Mata Hari/2.00=Lexibot.com
    MediaCrawler-1.0 (Experimental)=Mediacrawler.de
    MediaCrawler-1.0=Mediacrawler.de
    Mediapartners-Google/2.1 (+http://www.googlebot.com/bot.html)=Google.com
    Mediapartners-Google/2.1=Google.com
    MediaSearch/0.1=Www.fi
    MegaSheep v1.0 (www.searchuk.com internet sheep)=SearchUK.com
    MegaSheep v1.0=SearchUK.com
    Mercator=Altavista.com
    Mercator=Altavista.com
    Mercator-1.x=Altavista.com
    Mercator-2.0=Altavista.com
    Mercator-Scrub-1.1=Altavista.com
    metabot=metabot
    MFC_Tear_Sample=Microsoft.com
    MicrosoftPrototypeCrawler=Microsoft.com
    Minibot=Minibot
    MiracleAlphaTest=Nec.co.jp
    MnogoSearch/3.2.xx=mNoGoSearch.org
    moget/2.0=Goo.ne.jp
    moget/2.1 (moget@goo.ne.jp)=Goo.ne.jp
    moget/2.1=Goo.ne.jp
    moget@goo.ne.jp=Goo.ne.jp
    Mole2/1.0=Intags.de
    Morris - Mixcat Crawler (+http://mixcat.com)=Mixcat.com
    Morris - Mixcat Crawler=Mixcat.com
    Morris=Mixcat.com
    Mouse-House/7.4=Spidermonkey.ca
    mozDex/=Mozdex.com
    mozDex/0.04-dev=Mozdex.com
    Mozilla (Mozilla@somewhere.com)=Somewhere.com
    Mozilla/2.0 (compatible; Ask Jeeves)=Ask Jeeves
    Mozilla/2.0 (compatible; EZResult -- Internet Search Engine)=DirectHit.com
    Mozilla/3.0 (compatible; MuscatFerret/1.5.4; claude@euroferret.com)=Euroferret.com
    Mozilla/3.0 (compatible; MuscatFerret/1.5; olly@muscat.co.uk)=Euroferret.com
    Mozilla/3.0 (compatible; MuscatFerret/1.6.x; claude@euroferret.com)=Euroferret.com
    Mozilla/3.01 (Compatible; Links2Go Similarity Engine)=Links2go.com
    Mozilla/4.0 (compatible; FastCrawler3, support-fastcrawler3@fast.no)=AllTheWeb.com
    Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) TrueRobot; 1.5=Echo.com
    Mozilla/4.0 (compatible; www.galaxy.com)=Galaxy.com
    Mozilla/4.0 [en] (Ask Jeeves Corporate Spider)=Ask Jeeves
    Mozilla/4.0(compatible; Zealbot 1.0)=LookSmart.com
    Mozilla/4.0 (compatible; grub-client-2.x)=Grub.org
    MP3Bot=mp3bot.de
    MSNBOT/0.1=MSN.com
    msnbot=MSN.com
    MultiText/0.1=MultiText
    MultiText=MultiText
    MuscatFerret=Webtop.com
    NationalDirectoryAddURL/1.0=NationalDirectory.com
    NationalDirectory-SuperSpider=Nationaldirectory.com
    NationalDirectory-WebSpider/1.3=NationalDirectory.com
    Nazilla=Websmostlinked.com
    NEC Research Agent=NEC Research Agent
    NetCruiser=NetCruiser
    NetResearchServer(http://www.look.com)=Look.com
    NetResearchServer=Look.com
    NetZippy=Netzippy.com
    NextopiaBOT=Nextopia.com
    NG/1.0=Exabot.com
    Noago Spider=Noago.com
    NPBot=NameProtect.com
    NPBot-1/2.0=NameProtect.com
    nttdirectory_robot=Navi.ocn.ne.jp
    Nutch=Nutch.org
    NutchCVS/0.0x-dev=Nutch.org
    NutchOrg/0.0x-dev=Nutch.org
    NutchOrg=Nutch.org
    NZBot=NavigationZone.com
    ObjectsSearch/0.01=ObjectsSearch.com
    oBot ((compatible;Win32))=Cobion.com
    oBot=Cobion.com
    OliverPerry=Claymont.com
    OntoSpider/1.0 libwww-perl/5.65=Ontospider.i-n.info
    OntoSpider/1.0=Ontospider.i-n.info
    OntoSpider=Ontospider.i-n.info
    Openbot/=Openfind.com.tw
    Openbot/3.0+=Openfind.com.tw
    Openbot/3.0=Openfind.com.tw
    Opencola=Opencola.com
    Openfind piranha=Openfind.com
    Openfind Robot/1.1A2=Openfind.com.tw
    OpenTextSiteCrawler/2.9.2=Opentext.net
    Oracle Ultra Search=Oracle.com
    Oxxbot1=Oxxfordinfo.com
    Pansophica=Pansophica
    ParaSite/1.0b (http://www.ianett.com/parasite/)=Ianett.com
    ParaSite/1.0b=Ianett.com
    pd02_1.0.0 pd02_1.0.0@dzimi@post.sk=Post.sk
    peerbot=Peerbot.com
    Phoaks=Phoaks
    phortse@hanmail.net=Daum.net
    PICgrabber=PICgrabber
    PicoSearch/=PicoSearch.com
    PicoSearch/1.0=PicoSearch.com
    PictureOfInternet=PictureOfInternet
    PintaSpider=PintaSpider
    Pita=Pita Bot
    Pizilla++ ver 2.45=He.net
    PJspider/3.0 (pjspider@portaljuice.com; http://www.portaljuice.com)=Portaljuice.com
    PJspider/3.0=Portaljuice.com
    PJspider=Portaljuice.com
    polybot 1.0 (http://cis.poly.edu/polybot/)= PolyBot
    polybot 1.0=PolyBot
    PolyBot=PolyBot
    Pompos/1.x http://dir.com/pompos.html=Dir.com
    Pompos/1.x pompos@iliad.fr=Iliad.fr
    Pompos/1.x=Free.fr
    Popdexter/1.0=Popdex.com
    PortalBSpider/2.0 (spider@portalb.com)=Portalb.com
    PortalBSpider/2.0=Portalb.com
    Project XP5 [2.03.07-111203]= XP5 Bot
    Project XP5=XP5 Bot
    PROve AnswerBot 4.0=Answerchase.com
    psbot/0.1 (+http://www.picsearch.com/bot.html)=Picsearch.com
    psbot/0.1=Picsearch.com
    psbot=Picsearch.com
    Qango.com Web Directory (http://www.qango.com/)=Qango.com
    Qango.com Web Directory= Qango.com
    QPCreep Test Rig ( We are not indexing, just testing ) =Quepasa.com
    QPCreep Test Rig=Quepasa.com
    QuepasaCreep ( crawler@quepasacorp.com )=Quepasa.com
    QuepasaCreep v0.9.1x=Quepasa.com
    QuepasaCreep=Quepasa.com
    QueryN Metasearch=Queryn.com
    rabaz (rabaz at gigabaz dot com)=Brainbot.com
    Rabaz=Brainbot.com
    RaBot=Daum.net
    ramBot xtreme x.x=Intersearch.de
    Reaper [2.03.10-031204] (http://www.sitesearch.ca/reaper/)=Sitesearch.ca
    Reaper [2.03.10-031204]=Sitesearch.ca
    Reaper/2.0x (+http://www.sitesearch.ca/reaper)=Sitesearch.ca
    Reaper/2.0x=Sitesearch.ca
    Reaper=Sitesearch.ca
    roach.smo.av.com-1.0=Altavista.com
    roach.smo.av.com-1.0=Altavista.com
    RoboCrawl (www.canadiancontent.net)=CanadianContent.net
    RoboCrawl=CanadianContent.net
    Robot/www.pj-search.com=PJ-Search.com
    Robot@SuperSnooper.Com=SuperSnooper.com
    robot@xyleme.com=Xyleme.com
    robot-master@expressus.com=Expressus.com
    robot-response@openfind.com.tw=Openfind.com.tw
    Robozilla/1.0=Dmoz.org
    Rotondo/3.1 libwww/5.3.1=Qualigo.de
    ru-robot=Rutgers.edu
    Scooter/1.0 scooter@pa.dec.com=Altavista.com
    Scooter/1.0=Altavista.com
    Scooter/1.1 (custom)=Altavista.com
    Scooter/2.0 G.R.A.B. V1.1.0 =Altavista.com
    Scooter/2.0 G.R.A.B. X2.0=Altavista.com
    Scooter/3.3.QA.pczukor=Altavista.com
    Scooter/3.3.vscooter=Altavista.com
    Scooter/3.3_SF=Altavista.com
    Scooter/3.3=Altavista.com
    Scooter_bh0-3.0.3=Altavista.com
    Scooter_trk3-3.0.3=Altavista.com
    Scooter=Altavista.com
    Scooter2_Mercator_3-1.0=Altavista.com
    Scooter2_Mercator_x-x.0=Altavista.com
    Scooter-3.0.EU=Altavista.com
    Scooter-3.0.FS=Altavista.com
    Scooter-3.0.HD=Altavista.com
    Scooter-3.0.VNS=Altavista.com
    Scooter-3.0QI=Altavista.com
    Scooter-3.2.BT=Altavista.com
    Scooter-3.2.DIL=Altavista.com
    Scooter-3.2.EX=Altavista.com
    Scooter-3.2.JT=Altavista.com
    Scooter-3.2.NIV=Altavista.com
    Scooter-3.2.SF0=Altavista.com
    Scooter-3.2.snippet=Altavista.com
    Scooter-3.2=Altavista.com
    Scooter-3.3dev=Altavista.com
    Scooter-ARS-1.1=Altavista.com
    Scooter-ARS-1.1-ih=Altavista.com
    scooter-venus-3.0.vns=Altavista.com
    Scooter-W3.1.2=Altavista.com
    Scooter-W3-1.0=Altavista.com
    Scrubby/=Scrubtheweb.com
    search.at V1.2=Search.at
    search.ch V1.4.2=Search.ch
    search.ch V1.4=Search.ch
    SearchExpress Spider0.99=SearchExpress.com
    Searchit-Now Robot/2.2 (+http://www.searchit-now.co.uk)=Searchit-now.co.uk
    Searchit-Now Robot/2.2=Searchit-now.co.uk
    SearchSpider.com/1.1=Searchspider.com
    Searchspider/=Searchspider.com
    Searchspider/1.2=Searchspider.com
    sebastien.ailleret@inria.fr=Inria.fr
    Seeker.lookseek.com=Lookseek.com
    SeznamBot/1.0=Seznam.cz
    Shark=Openfind.com
    sherlock_spider=Sherlock.com.cn
    SightQuestBot/=Sightquest.com
    SiteSpider +(http://www.SiteSpider.com/)=Sitespider.com
    SiteSpider=Sitespider.com
    SiteXpert=Xtreeme.com
    Skampy/0.9.x= Skaffe.com
    Skimpy/0.x=Skaffe.com
    Slarp/0.1=Only.com
    Slider_Search_v1-de=Slider.com
    Slurp.so/1.0= Yahoo.com
    Slurp/2.0j=Yahoo.com
    Slurp/2.0-KiteHourly=Yahoo.com
    Slurp/2.0-OwlWeekly=Yahoo.com
    Slurp/3.0-AU=Yahoo.com
    slurp@inktomi.com=Yahoo.com
    slurp@inktomi= Yahoo.com
    Slurp=Yahoo.com
    SlySearch/1.0=Turnitin.com
    SlySearch/1.x=Turnitin.com
    SlySearch=Turnitin.com
    speedfind ramBot xtreme 8.1=Speedfind.de
    speedfind ramBot xtreme=Speedfind.de
    Speedy Spider=EntireWeb.com
    Speedy_Spider (http://www.entireweb.com)=EntireWeb.com
    Speedy_Spider=EntireWeb.com
    Spida/0.1=Only.com
    Spider TraficDublu=Traficdublu.ro
    Spider/maxbot.com admin@maxbot.com=Maxbot.com
    Spider/maxbot.com=Maxbot.com
    spider@aeneid.com=Yahoo.com
    Spider_Monkey/=Spidermonkey.ca
    spider_monkey=Spidermonkey.ca
    SpiderMonkey/7.x=Spidermonkey.ca
    Sqworm/2.9.81-BETA (beta_release; 20011102-760; i686-pc-linux-gnu)=Aol.com
    Sqworm=Aol.com
    Steeler/1.x (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)=Steeler Bot
    Steeler/1.x=Steeler Bot
    Steeler=Steeler Bot
    Suchbot=Suchbot.de
    Suchknecht.at-Robot=Suchknecht.at
    super-robot@super.navi.ocn.ne.jp=Navi.ocn.ne.jp
    support@canseek.ca=Canseek.ca
    Surfnomore Spider v1.1=Surfnomore.com
    SwissSearch V1.2=SwissSearch
    SynoBot=Synomia.fr
    TaWWWantula=TaWWWantula
    teoma_admin@hawkholdings.com=Teoma.com
    teoma_agent1=Teoma.com
    TeraCrawl=TeraCrawl
    Teradex_Mapper=Mapper.teradex.com
    The Informant=The Informant
    Toutatis 2.5-2=Hoppa.com
    Toutatis 3.3 (hoppa.com)=Hoppa.com
    Toutatis 4-19.6 (hoppa.com)=Hoppa.com
    TurnitinBot/x.x (http://www.turnitin.com/robot/crawlerinfo.html)=Turnitin.com
    TurnitinBot/x.x=Turnitin.com
    TurnitinBot=Turnitin.com
    Tutorial Crawler=Tutorgig.com
    Tv<nn>_Merc_resh_26_1_D-1.0=Altavista.com
    Tv<nn>_Merc_resh_26_1_D-1.0=Altavista.com
    TygoBot=Tygo.com
    UCmore=Ucmore.com
    UdmSearch=Mnogosearch.org
    UK Searcher Spider=Uksearcher.co.uk
    updated/0.1beta=Updated.com
    Ultraseek=Infoseek.com
    unlostBot=Unlost.com
    URLBlaze=Urlblaze.net
    UrlScope=UrlScope
    Vagabondo/1.x MT (webagent@wise-guys.nl)=Wise-guys.nl
    Vagabondo/2.0 MT (webagent at wise-guys dot nl)=Wise-guys.nl
    Vagabondo/2.0 MT (webagent@NOSPAMwise-guys.nl)=Wise-guys.nl
    Vagabondo/2.0 MT=Wise-guys.nl
    Vagabondo=Wise-guys.nl
    vspider=Verity.com
    webbandit/4.xx.0=WebBandit
    Webbandit=WebBandit
    Webclipping.com=WebClipping.com
    webcollage/1.xx=WebCollage.com
    webcollage=WebCollage.com
    WebCompass=WebCompass
    WebCrawler/3.0 Robot libwww/5.0a=WebCrawler
    WebCrawler-AddURL/2.0=WebCrawler
    WebFindBot(http://www.web-find.com)=Web-find.com
    WebFindBot=Web-find.com
    WebGather=WebGather
    WebGenie=WebGenie
    webmaster@intags.de=Intags.de
    whatUseek_winona/3.0=Whatuseek.com
    whatuseek=Whatuseek.com
    WhizBang! Lab=Whizbanglabs.com
    WIRE WebRefiner=Wire.co.uk
    wisenutbot=Looksmart.com
    WSCbot=WorldSearchCenter.com
    www.inktomisearch.com=Yahoo.com
    www.WebWombat.com.au=Webwombat.com.au
    Yandex=Yandex.com
    YottaShopping_Bot/4.12=YottaShopping.com
    Yellopet-Spider=YellowPet.com
    Zao/0.1 (http://www.kototoi.org/zao/)=Zao
    Zao/0.1=Zao
    Zao/0.2=Zao
    Zippy v2.0 - Zippyfinder.com=Zippyfinder.com
    ZyBorg=Wisenut.com
    Co-founder of Deepstash. Tech lover. Dinosaur. Dad. Chef.

Informații subiect

Utilizatori care navighează în acest subiect

Momentan este/sunt 1 utilizator(i) care navighează în acest subiect. (0 membrii și 1 vizitatori)

Thread-uri Similare

  1. robots.txt - ignorat de yahoo
    De B3aT în forumul Yahoo!
    Răspunsuri: 0
    Ultimul Post: 7th October 2008, 21:57
  2. robots.txt unreachable
    De alecs în forumul Google
    Răspunsuri: 0
    Ultimul Post: 3rd October 2007, 03:02
  3. robots.txt
    De cris în forumul Discutii generale privind optimizarea si motoarele de cautare
    Răspunsuri: 1
    Ultimul Post: 5th August 2007, 21:08
  4. robots.txt
    De brahma în forumul Discutii generale privind optimizarea si motoarele de cautare
    Răspunsuri: 2
    Ultimul Post: 31st July 2005, 01:25
  5. Răspunsuri: 12
    Ultimul Post: 12th June 2005, 10:23

Permisiuni postare

  • Nu puteţi posta subiecte noi.
  • Nu puteţi răspunde la subiecte
  • Nu puteţi adăuga ataşamente
  • Nu puteţi modifica posturile proprii
  •