What is Robots.txt?

Where the file has to live

A robots.txt file only works at the root of a host, as in https://example.com/robots.txt. A file at /blog/robots.txt is ignored. Each host, protocol and port counts as a separate site, so https://example.com and https://shop.example.com each need their own, and the http and https versions are treated separately too.

What robots.txt does not do

This is the misconception that causes the most damage. Robots.txt controls crawling, not indexing. If another site links to a page you have disallowed, Google can still index that URL and show it in results, usually with no description because it was never allowed to read the page. To keep a page out of search results, use a noindex tag or password protection, and leave the page crawlable so the instruction can be read.

Blocking a page also blocks everything on it

Disallowing a directory stops crawlers requesting the CSS, JavaScript and images inside it. That can prevent Google rendering the page as a visitor sees it, which affects how it is assessed. Blocking an image directory also removes those images from Google Images.

How to check yours

• Open the file directly in a browser at yourdomain.com/robots.txt and read it.
• Use the robots.txt report in Google Search Console to see the version Google last fetched and any errors it found.
• Test specific URLs against your rules before shipping a change, because a stray slash can disallow an entire site.
• Re-check it after any migration or replatform, since a staging file that disallows everything is a common and costly thing to ship.