Robots.txt tells search engine crawlers which paths they may request. Learn how it works, what it cannot do, and how to avoid…