What is robots.txt?

Robots.txt is a plain text file at the root of your site that tells search engine crawlers which paths they may or may not request. It is one of the simplest tools in technical SEO, and also one of the easiest to get wrong.

The key thing to understand is that robots.txt controls crawling, not indexing. It manages where bots go, not whether a page can appear in results. Google’s introduction to robots.txt documents the syntax and its limits.

Why robots.txt matters

On large sites, robots.txt helps steer crawl attention away from low-value areas like internal search results or admin paths, so crawlers spend more effort on pages that matter. This supports healthy crawlability.

The risk is that a single broad rule can block an entire section by mistake, cutting off pages you need indexed. Because the file is so powerful and so simple, small errors have large effects.

How robots.txt works

  • It uses User-agent lines to target specific crawlers and Disallow or Allow lines to set rules.
  • A disallowed path will not be crawled, but if it is already indexed or linked elsewhere, it can still appear in results without a snippet.
  • To keep a page out of the index, do not block it here. Allow crawling and use a noindex tag so engines can see and honor it.
  • You can reference your XML sitemap in robots.txt to aid discovery.

Robots.txt vs noindex

This is the most important distinction to remember.

  • Use robots.txt to manage crawling and crawl budget.
  • Use noindex to keep a page out of the index. This requires the page to be crawlable, so do not block it in robots.txt at the same time.

Blocking a page in robots.txt when you actually want it removed from the index is a common and costly mistake, because the engine cannot crawl the page to see the noindex signal.

Common robots.txt mistakes

  • Disallowing a folder that contains pages you want indexed.
  • Trying to remove a page from search with a disallow rule alone.
  • Blocking CSS or JavaScript that the page needs to render correctly.
  • Leaving a site-wide disallow in place after launch, left over from development.

Key takeaways

  • Robots.txt controls crawling, not indexing.
  • To remove a page from the index, allow crawling and use noindex, not a disallow rule.
  • Test changes carefully, because one broad rule can hide an entire section.

Frequently asked questions

Will robots.txt remove a page from Google?
Not reliably. A disallowed page can still be indexed if it is linked elsewhere. Use a noindex tag on a crawlable page to remove it.

Should I block CSS and JavaScript?
No. Search engines need those files to render and understand the page. Blocking them can hurt how your pages are evaluated.

Part of our Technical SEO guide. For help, see our Technical SEO service.