Use this tool
to generate a simple robots.txt file for your website. This file allows
you to hide files or directories that you don’t wish the search engine
spiders to find. This generator tool is designed to create the text for
the file for you; you can then make changes afterward should you deem it
necessary.
Generating a robot file with this tool is
ideal if you wish to block certain directories or files from search
engines.
To use the generator tool, enter the
required information below and click the button. You will then be shown
the text for the file. Copy this to a file called robots.txt and place on
the root of your website (in the same place as your home page).
Imposing Restrictions
You may impose restrictions on which web pages to disallow indexing. By
default, most users will want to allow all directories except their /cgi-bin/
directory, which commonly holds scripts, and their images directory
/images/. To enable all web pages, select Yes to "Enable All Webpages,"
then enter each web page or directory path in the exclusion box, one per
line.
Example: "http://www.yourdomain.com/cgi-bin/" (Excludes the /cgi-bin/
directory)
Example: "http://www.yourdomain.com/images/" (Excludes the
/images/ directory)
Example: "http://www.yourdomain.com/welcome.html" (Excludes the
/welcome.html web page)
