The robots.txt file is used to restrict internet search engine crawlers from accessing sections of your web site. Even though the file is extremely practical, It is also a straightforward method to inadvertently block crawlers. Lots of in the Web optimization industry have believed this was the case for https://www.youtube.com/watch?v=EVeTP0aV8gY