# # $Header: robots.txt 115.0 2004/08/26 07:43:47 kroychow ship $ # ############################################################### # # This file is automatically generated by AutoConfig. It will be read and # overwritten. If you were instructed to edit this file, or if you are not # able to use the settings created by AutoConfig, refer to Metalink document # 165195.1 for assistance. # # The robots.txt is used by many search engines (however not all) to limit inclusion into their databases. Web crawlers look for a robots.txt file in the web server root directory (i.e., http://sun.example.com/robots.txt). The robots.txt should contain the following lines, which will stop most web crawlers from looking at any pages on the server . User-agent: discobot Disallow: / # # ############################################################### # Disallow wellbehaved webcrawlers from indexing # http://v490a.specialized.com User-agent: * Disallow: /zz/