Crawler-Bots-Spiders

Crawler, Bots and Spider visit your web site every day. Major search engines use them and others do too. This category will provide you with more details about a lot of them so you can decide which ones you want on your web site and whom you rather deny access.

The robots.txt file can be used to disallow access for a bot. I found it very difficult to update the robots.txt file with the new bots as I don't like to block everyone but the known search engines. Our list of crawler, bots and spiders should help you to at least know some bot names.

I think it is true to say that the number of bots visitng your site can give you an idea about the visibility of your site on the internet. I mean, more bots means more popular site. It's not all bad. :-)

Bookmark this page by using:

• Delicious
• Digg
• reddit
• Facebook
• StumbleUpon
• Google

These sites allow you to store, tag and share links across the internet to share with your friends and family. The links can also be accessed again by you from any PC.

When you find a page on our site of interest to you, which you wish to share or use again yourself in the future, just click on one of these links to add the page to your list.

These site all require you to register but are free to use. Once registered you can start bookmarking.

• Secure Password Manager
• My Favourite Firewall
• Free FTP Client