It is used to include directories of your websites for crawling and indexing for search engine and access for users. Sitemap is a xml file which contain full list of your website urls. Remember robots.txt instruction to restrict access to your sensitive information of your page is not formidable on web page security ground. Write it properly including your content enriched pages and other public pages and exclude any pages which contain sensitive information. So add a robots.txt file in your website root directory. Your website directories will be crawled and indexed on search engine according to robots.txt instructions.
A full access or a full restriction or customized access or restriction can be imposed through robots.txt.
robots.txt contains the search bots or others bots name, directory list allowed or disallowed to be indexing and crawling for bots, time delay for bots to crawl and indexing and even the sitemap url. Robots.txt is text file that reside on website root directory and contains the instruction for various robots (mainly search engine robots) for how to crawl and indexing your website for their webpage.
0 kommentar(er)
