Robots.txt is a file that exists on the root Listing of each Web-site and can be used to instruct search engines like google on which directories/information of the web site they're able to crawl and include things like in their index. Not surprisingly, they don’t expose all of the algorithm https://news.rafeeg.ae