Faster Google Crawl Rate

January 22, 2008 by Collin LaHay
Filed under: General 

Let me start off saying that if you do not use Google Webmaster Tools, you should start trying it out. It is a great tool that shows you near-instant updates of backlinks to your posts, how often Google views your blog, as well as any errors that the Google bot finds.

Utilizing the tools will help you keep your blog search engine friendly and clean. Now for the point of this article, Google normally has a “normal” and a “slower” option for how often it will crawl your page. If you write at least a few times a week you should keep this at “normal,” but if you have a slow webhost or another reason that you do not want Google visiting a lot you can set it to slow.

crawlrates1.jpg

Just in the last few days, Google has deemed my blog worthy of a “Faster” crawl rate. I enabled the option yesterday and Google viewed my blog 270 times when the daily average is 70. With this new faster crawling, my blog posts are getting spidered, indexed, and cached in a matter of hours… sometimes even within minutes.

crawl1.jpg

If you have been blogging for awhile, I recommend checking out the crawl rate in your webmaster tools to see if you can enable the option. Since Google barely uses any bandwidth at all, I highly recommend using the Faster rate if your hosting can support it. In my opinion, the more times Google browses my blog the better!

Do any of you have a faster crawl rate option?

Comments

24 Comments on Faster Google Crawl Rate

  1. SEO Meter on Tue, 22nd Jan 2008 4:19 pm

    2. “Since Google barely uses any bandwidth at all, I highly recommend using the Faster rate if your hosting can support it.”

    That’s because the number of Google-indexed pages for your blog is small. For larger sites, it will make much difference in terms of bandwidth consumption. So care needs to be taken before turning its switch.

  2. Collin LaHay on Tue, 22nd Jan 2008 4:57 pm

    3. While this is true, bandwidth is cheap and is only getting cheaper. If you have a site that is fifty times as big as my blog then you should be getting crawled well already. Most hosting companies provide terabyte(s) of bandwidth now, so a few hundred kilobytes, or even a few megabytes do little damage to the bill. =)

  3. Sudarshan on Tue, 22nd Jan 2008 6:55 pm

    4. I am going to use this option to my celebrity blog an let u know on how it goes

  4. Eric on Wed, 23rd Jan 2008 2:27 am

    5. My blog got the option available in August 2007, just a month after I’ve started my blog.

    If your site updates a lot (like a blog should do) you definately need to activate this option. It helps getting your pages indexed faster. Which could result in high rankings in Google search.

  5. Ethan Christ on Wed, 23rd Jan 2008 3:00 am

    6. Thanks for the tip. Going to implement a faster crawl rate for Slam Blogger as we speak.

  6. Binh Nguyen on Wed, 23rd Jan 2008 6:58 am

    7. I have faster crawl rate. ;)

  7. Tyler on Wed, 23rd Jan 2008 9:45 am

    8. I have seen firsthand what raising that option can do to your web server. Sometimes Google will turn that option off so you cant select it.

  8. James Pong on Wed, 23rd Jan 2008 11:23 am

    9. I have seen Google indexes my page within 40 minutes after posting it. Does the crawl rate goes “Faster” by default? Thanks for the tip. I’ll try to check the webmaster tools more often.

  9. Steve on Wed, 23rd Jan 2008 12:31 pm

    10. Fastest I’m now getting is inclusion within 5 minutes as a Google News Source. It is both a blessing and a curse. Sucks when you find the type just a minute too late.

  10. Leandro on Wed, 23rd Jan 2008 1:04 pm

    11. thanks for the tip. I use the google sitemap for a long time and never saw this tool.

  11. rhyan on Wed, 23rd Jan 2008 2:50 pm

    12. thanks for this tip. I will take a look at this one :D

  12. Sunglasses Auctions on Thu, 24th Jan 2008 7:34 pm

    13. Collin,

    When i was browsing around the google webmaster tools, i noticed in the Tools section something called ‘Enable enhanced image search’ .

    Do you know if this is a good or bad thing to activate? i did a search but found mixed results

    Cheers

  13. Catherine Lawson on Thu, 24th Jan 2008 8:03 pm

    14. Thanks for the great tip. There’s certainly no harm in trying it to see what happens.

  14. Collin LaHay on Thu, 24th Jan 2008 11:01 pm

    15. I saw this too. I enabled it weeks ago, but I cannot give you an answer. I have not received any traffic from Google images so I have no idea how well it works.

    I would assume enabling it is better than not enabling it. =)

  15. arlene on Fri, 25th Jan 2008 12:42 am

    16. thanks for this information. Am playing on it now but my baby is still a baby. has not started crawling yet. hehehe

    good day!

  16. SEOMeter graphs your Google crawl rate | Make Money Blogging on Sun, 27th Jan 2008 2:17 pm

    17. [...] it myself. I wish I would have joined a month ago to see an accurate growth after switching to the faster google crawl option in my webmaster [...]

  17. Karel Geenen on Wed, 30th Jan 2008 4:54 pm

    18. My Dutch blog doesn’t have the faster crawl option. However my posts get indexed within 10 minutes, always.

    My advice is to check first how long it takes before a post get’s indexed. If it takes to long, activate the faster crawl. If it gets indexed within the hour, leave it as it is.

  18. paulette on Thu, 31st Jan 2008 11:19 am

    19. Thanks for the info. Google really is a fast tool for searching

  19. Bolkie on Thu, 7th Feb 2008 6:48 am

    20. Does this option work for established blogs only, or also for newer/fresh blogs?

  20. Collin LaHay on Thu, 7th Feb 2008 7:40 am

    21. Blogs that Google thinks should be crawled more often. The only way to find out is to check Webmaster Tools.

  21. iGuide on Sat, 10th Jan 2009 5:52 pm

    22. And what if your site DOESN’T update that frequently, say only once a month? Then, would it actually hurt you to set your settings to “faster” and let the bot discover your pages don’t change that often? I already see Google and Yahoo are each now pulling 3-4 gigsabytes a month, plus another 2 or 3 gigabytes for MSN. That sounds like a lot, but all spiders put together still account for lass than half of traffic.

  22. Collin LaHay on Sat, 10th Jan 2009 7:17 pm

    23. The spiders visiting more month will never hurt you, but it will use a lot more bandwidth which may be a problem for some people.

  23. lee on Fri, 11th Sep 2009 3:02 pm

    24. hi i have been waiting for google and bing to index my blog http://www.diyanswerdirect.com/blog for two weeks now and am getting very frustrated with them. aswell as iam still waiting for them to update my site http://www.diyanswerdirect.com. My question is when do they update and how can i get them to crawl or spider my site.

  24. Imran on Mon, 21st Dec 2009 10:43 pm

    25. Make sure in your blog settings your blog is not set to ‘private’ (which I think is default if you use wordpress)

Tell me what you're thinking...
and oh, if you want a pic to show with your comment, go get a gravatar!