In Digital Pacific’s most recent survey, there was a suggestion that we provide every customer with a robots.txt file in their root directory. We understand this would be a plus for all of our customers, but it would be a very large feat for us to undertake. However, it will please most of you to know that creating a robots.txt file of your own is a fairly simple task that can be completed with the following guide.
In case you are not sure why you might need a robots.txt file, it is actually there to tell search engine robots which pages they should crawl in order to index. So, if there are pages that you would prefer to NOT have indexed (e.g. secure pages, such as login pages, or test pages), then you can simply put in a line to “disallow” the search engines to not waste their time crawling them.
Creating a Robots.txt File
Open up notepad on your computer and insert the following lines into the document:
User-agent: *
Disallow:
By using this format above, you are giving a wildcard (*) to user agents, meaning the file is targeting any search robot. By leaving the Disallow option blank, you are letting any search robot crawl and index all pages on your website.
If you want to keep the robots from crawling specific pages, such as login pages or test pages, then add those to the Disallow section:
User-agent: *
Disallow: /login/
Disallow: /testpage/
Completing the Process
Once you are happy with your file, make sure you save it as a plain text document so it appears as robots.txt. Upload the file to your root directory on your server and use a free robots.txt validation program, like this one, to verify the syntax is in correct working order.
Congratulations! You now have your very own robots.txt file!
