Robotst.txt are a wonderful tool that you can use to instruct search engine crawlers how you would want it to crawl your website. They are used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google. To keep a web page out of Google, block indexing with noindex or password-protect the page.
Although robots.txt isn’t a the most powerful tool at your disposal, it can at least assist in preventing your site from being skipped over by the crawlers.
Dan Taylor shows us the most common problems with robots.txt files, as well as the impact they might have on your website and it’s presence on Google’s search engine. In his column, Dan gives us the information we need to fix issues if you think this has happened.