What is a robots.txt File?
A robots.txt file is a plain text file that provides information to search engine spiders that come to your site. Most search engines today look for a robots.txt file before they do anything else. Check your stats, and you may see a bunch of "file not found" errors caused by the spiders looking for robots.txt.
That, in itself, is reason enough to have one.
You may have information or web pages in certain directories that you do not want indexed by the search engines, you know, like the download page for products you are selling!!!
You may have duplicate pages that you use for various advertising programs, and you certainly don't want the search engines to think you are spamming by having numerous copies of the same webpage.
How Do You Create a robots.txt file?
Use a text document program such as Notepad. Use the following format for creating your file.
User-agent: *
Disallow: /insert directory name/
Disallow: /insert directory name/
The * by User-agent is your direction that NO search engine spider should index the disallowed directories. The / behind the directory name indicates that no file inside the directory is to be indexed.
Once you have completed your robots.txt file, you should upload it to your web server in ASCII mode.