Use Robots.txt, Save the World
No commentsRobots.txt Help the Search Engines Learn All About Your Website
There is a growing interest in the little known file that every website should have in the root directory: robots.txt
It’s a very simple text file you can find all about at the robotstxt.org website.
Why should you use it ? Here are some good reasons for you to consider.
Controlled Access to Your Content
With a robots.txt file you can “ask” the search engines to “keep out” of certain areas of your website. A typical area you might like to exclude is your images folder: If you aren’t a photographer, painter and your images are for your website use only, there are good chances you don’t want them to be indexed and showing up on image search engines, for people to download, or hotlink.
Unfortunately grabbers and similar software (such as Email harvesting applications) will not read your robots.txt file disregarding any indication you may provide in this respect. But that’s life isn’t it, always someone being disrespectful to say the least …
You can keep search engines away from content you wish to keep out of sight, but remember your robots file is also subject to attention of hackers seeking sensitive objectives you might inadvertently lÃst: keeping out the robots while inviting the hackers � keep this in mind.
The Growing Importance of Robots.Txt
At SES New York a robots.txt summit was held where major search engines (Ask, Google, Microsoft, Yahoo!) participated, sharing interesting information on this file. Here are some numbers.
According to Keith Hogan from Ask:
i) Less than 35% of websites have a robots.txt file
ii) The majority of robots.txt files are copied from others found online
iii) On many occasions robots.txt files are provided by your web hostÃng service
It looks like the majority of webmasters aren’t familiar with this file. This is going to play a major role as the size of the web continues to grow: Spidering is a costly effort that search engines tend to optimize. Those web sites demonstrating optimal command (which in turn determines efficiency) will be rewarded.
Wednesday, April 25th, 2007 at 9:03 am and is filed under SEO, SEO Tips and Techniques. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.
