Robots.txt is actually a file that exists on the foundation directory of each Site and can be used to instruct search engines like google and yahoo on which directories/information of the website they are able to crawl and include in their index. When crawling a webpage, they figure out its https://news.rafeeg.ae