Create/Optimize robots.txt file

We will create a robots.txt file if one does not exist for your domain, if you have a robots.txt file in place we will optimise the content of the file to have the site engines only crawl the necessary pages.