Robots.txt<\/a> is a plain text file at the root of your site that tells search engine crawlers which paths they may or may not request. It is one of the simplest tools in technical SEO<\/a>, and also one of the easiest to get wrong.<\/p>\n The key thing to understand is that robots.txt controls crawling, not indexing. It manages where bots go, not whether a page can appear in results. Google’s introduction to robots.txt<\/a> documents the syntax and its limits.<\/p>\n On large sites, robots.txt helps steer crawl attention away from low-value areas like internal search results or admin paths, so crawlers spend more effort on pages that matter. This supports healthy crawlability<\/a>.<\/p>\n The risk is that a single broad rule can block an entire section by mistake, cutting off pages you need indexed. Because the file is so powerful and so simple, small errors have large effects.<\/p>\n This is the most important distinction to remember.<\/p>\n Blocking a page in robots.txt when you actually want it removed from the index is a common and costly mistake, because the engine cannot crawl the page to see the noindex signal.<\/p>\n Will robots.txt remove a page from Google?<\/strong> Should I block CSS and JavaScript?<\/strong> Part of our Technical SEO guide<\/a>. For help, see our Technical SEO service<\/a>.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":" Robots.txt tells search engine crawlers which paths they may request. Learn how it works, what it cannot do, and how to avoid costly mistakes.<\/p>\n","protected":false},"author":1,"featured_media":2915,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"gallery","meta":{"rank_math_focus_keyword":"robots.txt","rank_math_title":"Robots.txt: How to Control What Search Engines Crawl | 2tentech","rank_math_description":"Robots.txt tells crawlers which paths they may request. Learn how it works, what it cannot do, and how to avoid costly mistakes.","footnotes":""},"categories":[1],"tags":[175,174,163],"class_list":["post-2878","post","type-post","status-publish","format-gallery","has-post-thumbnail","hentry","category-seo","tag-crawl-budget","tag-robots-txt","tag-technical-seo","post_format-post-format-gallery"],"_links":{"self":[{"href":"https:\/\/2tentech.com\/blog\/wp-json\/wp\/v2\/posts\/2878","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/2tentech.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/2tentech.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/2tentech.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/2tentech.com\/blog\/wp-json\/wp\/v2\/comments?post=2878"}],"version-history":[{"count":2,"href":"https:\/\/2tentech.com\/blog\/wp-json\/wp\/v2\/posts\/2878\/revisions"}],"predecessor-version":[{"id":2895,"href":"https:\/\/2tentech.com\/blog\/wp-json\/wp\/v2\/posts\/2878\/revisions\/2895"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/2tentech.com\/blog\/wp-json\/wp\/v2\/media\/2915"}],"wp:attachment":[{"href":"https:\/\/2tentech.com\/blog\/wp-json\/wp\/v2\/media?parent=2878"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/2tentech.com\/blog\/wp-json\/wp\/v2\/categories?post=2878"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/2tentech.com\/blog\/wp-json\/wp\/v2\/tags?post=2878"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Why robots.txt matters<\/h2>\n
How robots.txt works<\/h2>\n
\n
User-agent<\/code> lines to target specific crawlers and Disallow<\/code> or Allow<\/code> lines to set rules.<\/li>\nRobots.txt vs noindex<\/h2>\n
\n
Common robots.txt mistakes<\/h2>\n
\n
Key takeaways<\/h2>\n
\n
Frequently asked questions<\/h2>\n
\nNot reliably. A disallowed page can still be indexed if it is linked elsewhere. Use a noindex tag on a crawlable page to remove it.<\/p>\n
\nNo. Search engines need those files to render and understand the page. Blocking them can hurt how your pages are evaluated.<\/p>\n
\n