Atacat de crawlerii Yahoo! Ajutor!

[meetzah]

Asa ceva e imposibil! Am incercat sa il limitez din robots.txt (nu sa interzic definitiv, ci doar sa maresc perioada dintre vizite) si tot nu pot sa scap. Ia uitati-va si voi ce imi arata logurile pe ziua de azi (adica nu pe toata ziua, ci doar la momentul la care am solicitat raportul) referitor la crawlerii Yahoo:

74.6.28.234 lj511076.crawl.yahoo.net
74.6.28.143 lj511089.crawl.yahoo.net
74.6.28.151 lj511097.crawl.yahoo.net
74.6.28.156 lj511102.crawl.yahoo.net
74.6.28.171 lj511117.crawl.yahoo.net
74.6.28.102 lj511122.crawl.yahoo.net
74.6.28.121 lj511141.crawl.yahoo.net
74.6.28.70 lj511147.crawl.yahoo.net
74.6.28.22 lj511172.crawl.yahoo.net
74.6.28.44 lj511194.crawl.yahoo.net
74.6.27.105 lj511205.crawl.yahoo.net
74.6.27.111 lj511211.crawl.yahoo.net
74.6.27.72 lj511229.crawl.yahoo.net
74.6.26.202 lj511284.crawl.yahoo.net
74.6.26.146 lj511332.crawl.yahoo.net
74.6.26.155 lj511341.crawl.yahoo.net
74.6.26.165 lj511351.crawl.yahoo.net
74.6.26.104 lj511364.crawl.yahoo.net
74.6.26.76 lj511393.crawl.yahoo.net
74.6.26.46 lj511436.crawl.yahoo.net
74.6.25.207 lj511449.crawl.yahoo.net
74.6.25.210 lj511452.crawl.yahoo.net
74.6.25.214 lj511456.crawl.yahoo.net
74.6.25.120 lj511540.crawl.yahoo.net
74.6.25.78 lj511555.crawl.yahoo.net
74.6.25.23 lj511573.crawl.yahoo.net
74.6.24.205 lj511607.crawl.yahoo.net
74.6.24.206 lj511608.crawl.yahoo.net
74.6.24.233 lj511635.crawl.yahoo.net
74.6.24.136 lj511642.crawl.yahoo.net
74.6.24.171 lj511677.crawl.yahoo.net
74.6.24.27 lj511737.crawl.yahoo.net
74.6.23.206 lj511768.crawl.yahoo.net
74.6.23.221 lj511783.crawl.yahoo.net
74.6.23.142 lj511808.crawl.yahoo.net
74.6.23.123 lj511863.crawl.yahoo.net
74.6.23.16 lj511886.crawl.yahoo.net
74.6.22.209 lj511931.crawl.yahoo.net
74.6.22.219 lj511941.crawl.yahoo.net
74.6.22.228 lj511950.crawl.yahoo.net
74.6.22.46 lj511996.crawl.yahoo.net
74.6.20.205 lj512007.crawl.yahoo.net
74.6.20.208 lj512010.crawl.yahoo.net
74.6.20.218 lj512020.crawl.yahoo.net
74.6.20.230 lj512032.crawl.yahoo.net
74.6.20.233 lj512035.crawl.yahoo.net
74.6.20.164 lj512070.crawl.yahoo.net
74.6.20.101 lj512081.crawl.yahoo.net
74.6.20.111 lj512091.crawl.yahoo.net
74.6.74.124 lj611054.crawl.yahoo.net
74.6.74.6 lj611096.crawl.yahoo.net
74.6.74.46 lj611108.crawl.yahoo.net
74.6.73.165 lj611197.crawl.yahoo.net
74.6.73.90 lj611212.crawl.yahoo.net
74.6.73.47 lj611272.crawl.yahoo.net
74.6.72.227 lj611283.crawl.yahoo.net
74.6.72.8 lj611400.crawl.yahoo.net
74.6.71.187 lj611493.crawl.yahoo.net
74.6.71.162 lj611500.crawl.yahoo.net
74.6.70.75 lj611706.crawl.yahoo.net
74.6.69.139 lj611855.crawl.yahoo.net
74.6.69.14 lj611938.crawl.yahoo.net
74.6.69.49 lj611952.crawl.yahoo.net
74.6.68.167 lj611997.crawl.yahoo.net
74.6.68.42 lj612090.crawl.yahoo.net
74.6.68.21 lj612114.crawl.yahoo.net
74.6.67.201 lj612131.crawl.yahoo.net
74.6.67.160 lj612168.crawl.yahoo.net
74.6.67.143 lj612172.crawl.yahoo.net
74.6.67.111 lj612218.crawl.yahoo.net
74.6.75.29 lj612515.crawl.yahoo.net
74.6.74.212 lj612570.crawl.yahoo.net
74.6.74.231 lj612590.crawl.yahoo.net
74.6.21.140 rz502206.inktomisearch.com
74.6.21.143 rz502209.inktomisearch.com
74.6.21.144 rz502210.inktomisearch.com
74.6.21.154 rz502220.inktomisearch.com
74.6.21.159 rz502225.inktomisearch.com
74.6.21.166 rz502232.inktomisearch.com
74.6.21.168 rz502234.inktomisearch.com
74.6.21.170 rz502236.inktomisearch.com
74.6.21.104 rz502244.inktomisearch.com
74.6.21.108 rz502248.inktomisearch.com
74.6.21.118 rz502258.inktomisearch.com
74.6.21.120 rz502260.inktomisearch.com
74.6.21.123 rz502263.inktomisearch.com
74.6.21.125 rz502265.inktomisearch.com
74.6.21.75 rz502272.inktomisearch.com
74.6.21.79 rz502276.inktomisearch.com
74.6.21.14 rz502284.inktomisearch.com
74.6.21.15 rz502285.inktomisearch.com
74.6.21.29 rz502299.inktomisearch.com
74.6.21.30 rz502300.inktomisearch.com
74.6.21.38 rz502308.inktomisearch.com
74.6.21.43 rz502313.inktomisearch.com
74.6.21.49 rz502319.inktomisearch.com
74.6.19.10 rz502320.inktomisearch.com
74.6.19.20 rz502330.inktomisearch.com
74.6.19.23 rz502333.inktomisearch.com
74.6.19.30 rz502340.inktomisearch.com
74.6.19.32 rz502342.inktomisearch.com
74.6.19.35 rz502345.inktomisearch.com
74.6.18.202 rz502364.inktomisearch.com
74.6.18.205 rz502367.inktomisearch.com
74.6.18.206 rz502368.inktomisearch.com
74.6.18.210 rz502372.inktomisearch.com
74.6.18.213 rz502375.inktomisearch.com
74.6.18.221 rz502383.inktomisearch.com
74.6.18.222 rz502384.inktomisearch.com
74.6.18.232 rz502394.inktomisearch.com
74.6.18.233 rz502395.inktomisearch.com
74.6.18.235 rz502397.inktomisearch.com
74.6.18.135 rz502401.inktomisearch.com
74.6.18.137 rz502403.inktomisearch.com
74.6.18.138 rz502404.inktomisearch.com
74.6.18.140 rz502406.inktomisearch.com
74.6.18.142 rz502408.inktomisearch.com
74.6.18.143 rz502409.inktomisearch.com
74.6.18.144 rz502410.inktomisearch.com
74.6.18.103 rz502443.inktomisearch.com
74.6.18.114 rz502454.inktomisearch.com
74.6.18.115 rz502455.inktomisearch.com
74.6.18.121 rz502461.inktomisearch.com
74.6.18.126 rz502466.inktomisearch.com
74.6.18.73 rz502470.inktomisearch.com
74.6.18.79 rz502476.inktomisearch.com
74.6.18.10 rz502480.inktomisearch.com
74.6.18.11 rz502481.inktomisearch.com
74.6.18.13 rz502483.inktomisearch.com
74.6.18.22 rz502492.inktomisearch.com
74.6.18.24 rz502494.inktomisearch.com
74.6.18.34 rz502504.inktomisearch.com
74.6.18.37 rz502507.inktomisearch.com
74.6.18.41 rz502511.inktomisearch.com
74.6.18.47 rz502517.inktomisearch.com
74.6.18.48 rz502518.inktomisearch.com
74.6.17.201 rz502523.inktomisearch.com
74.6.17.210 rz502532.inktomisearch.com
74.6.17.220 rz502542.inktomisearch.com
74.6.17.224 rz502546.inktomisearch.com
74.6.17.136 rz502562.inktomisearch.com
74.6.17.146 rz502572.inktomisearch.com
74.6.17.147 rz502573.inktomisearch.com
74.6.17.150 rz502576.inktomisearch.com
74.6.17.151 rz502577.inktomisearch.com
74.6.17.157 rz502583.inktomisearch.com
74.6.17.161 rz502587.inktomisearch.com
74.6.17.163 rz502589.inktomisearch.com
74.6.17.165 rz502591.inktomisearch.com
74.6.17.172 rz502598.inktomisearch.com
74.6.17.103 rz502603.inktomisearch.com
74.6.17.106 rz502606.inktomisearch.com
74.6.17.111 rz502611.inktomisearch.com
74.6.17.112 rz502612.inktomisearch.com
74.6.17.113 rz502613.inktomisearch.com
74.6.17.125 rz502625.inktomisearch.com
74.6.17.73 rz502630.inktomisearch.com
74.6.17.75 rz502632.inktomisearch.com
74.6.17.76 rz502633.inktomisearch.com
74.6.17.78 rz502635.inktomisearch.com

Imi mananca banda in prostie. Are cineva o solutie? Ca daca nu rezolv nimic, banez deinitiv Yahoo din robots.txt si termin treaba. Oricum, numarul de vizitatori pe care mi-l aduce este nesemnificativ. Totusi, asa ceva nu este normal si refuz sa cred ca se intample pe fiecare site.

P.S. asta in conditiile in care am banat deja 60 de IP-uri ca cele de mai sus.....

[Krumel]

E un update de Yahoo!, am postat pe forum.
Recupereaza baietii.

[Marius Mailat]

La mine o dat nebunia peste ask.com. Do not ask why. Indexeaza intro nebunie.

[meetzah]

O fu update, nu zic nu, dar chiar asa la nesimtire sa se puna pe siturile oamenilor. Hai ca la cine are limita de trafic mare e ok, dar la celelalte situri? Eram curios si daca pe alte situri e la fel in perioada asta.

[Marius Mailat]

La ine s-a simtit peste tot. Ask a venit in timpul zilei, yahoo noaptea.

[meetzah]

Bun... si eu cum scap pana la urma de botii astia? nu are nimeni nici o idee (in afara de bann total) ?

[Krumel]

Robots.txt si opreste robotii Yahoo! o perioada.

[meetzah]

multzam, dar am specificat ca nu vreau sa ii opresc deloc. poate e ceva la situl meu care ii face sa ma invadeze asa... oare sa fie faptul ca pe site se adauga zilnic continut nou?

[Krumel]

Ti-am zis: eu de ceva timp ii am pe site zilnic.