Making your robots.txt.
Robots.txt is a simple file that you can edit in Notepad and after save in the root directory of your site, where you also have your home page. Each entry for your robots.txt has to carry two lines: User-Agent: [Spider/ Boot name] Disallow: [Directory/ File Name]
According to your understanding for using robots.txt, here is how you will be able to put it together:
1. If you need to boot out a file from an individual Search Engine:
User-Agent : Googlebot Disallow: /private/privatefile.htm
2. If you need to boot out only a section of your site from spiders:
You don't need to assign each robot that you wish to boot out anymore, because you can simply use a wildcard character, ‘*’, which will conduce to its quick exclusion. User-Agent: * Disallow: /newsection/
3. If you need to grant search engine spiders to index everything:
Using the same wildcard, ‘*’, you indicate that all spiders are welcome to crawl your entire site. Remember you want to leave the second line, disallow, empty, i.e. your prohibit from nowhere.
User-agent : * Disallow: formerly you got your robots.txt in place, you need to make certain they've been done correctly. The tool to use for this is Google Webmaster Tool. As we have discussed in a previous blog (Create your Search Engine Marketing schemes by Using Google Webmaster Tool), one of the characteristics provided by this tool grants you to assure your robots.txt. Fundamentally, Google will automatically and at once retrievethe robots.txt from your website and adds the main url to the list of urls that should be assured.
If you want to share your experience in using robots.txt, Then just Contact Us for More about
Best Robots.txt Creation Services.
Shah Net Technologies Pvt Ltd is a Well-Known SEO company, online reputation management and SEM company. Click here to
contact us. for your Website 's Robot.txt Creation to Crawl your site to major Search Engines.
