Metashit

For most of us, search engine robots such as Googlebot indexing our site TOO quickly is not a problem. But what if your domain has 50,000 pages and your webserver cannot handle requests as quickly as Googlebot demands? Well, here are a few solutions:

1. Add the following code to your robots.txt file, which instructs robots to wait 5 seconds between each request (the default is 1 second):
   

   User-agent: *
   Crawl-delay: 5

2. Sign up for Google Webmaster Tools and change the crawl rate to slower.
   

   

3. As a last resort, you can block robots altogether. This is not recommended since you will not be indexed, but it may be necessary if your webserver cannot handle the bandwidth:
   

   User-agent: *
   Disallow: /

Hopefully this will not even be an issue if you are using a good webhost. I recommend getting a dedicated server from Dreamhost.