Robots.txt: Just like a stoplight, a robots.txt file gives online search engine spiders or crawlers way. You’ll use this file to allow spiders to crawl your site. You may also use it to halt them from browsing URLs like your paid landing pages. How will you get All those hyperlinks? https://www.youtube.com/watch?v=7cBHAtTOxoI