Yahoo! Slurp crawl delay: Reduce the number of requests that Yahoo! Slurp crawler makes to your site. I wish Google had something of similar nature since I almost get DDoS’d at time by the Google crawlers (multiple crawlers seem to hit the site at the same time, requesting every article in rapid succession). Not that I am complaining that Google crawlers visit this blog. 
(No Ratings Yet)
Loading ...
