post-page

Yahoo! Slurp crawl delay

No
responses
by
 
on
February 11th, 2005
in
Cool Scripts, LinkyLoo

Yahoo! Slurp crawl delay: Reduce the number of requests that Yahoo! Slurp crawler makes to your site. I wish Google had something of similar nature since I almost get DDoS’d at time by the Google crawlers (multiple crawlers seem to hit the site at the same time, requesting every article in rapid succession). Not that I am complaining that Google crawlers visit this blog. ;)

heading
No
Responses

 

Comments are closed.

Obviously Powered by WordPress. © 2003-2013

page counter
css.php