Robots.txt is a text file at a site's root that tells search engine crawlers which paths they may or may not request.
Also known as: Robots File
It manages crawl access and budget, but it controls crawling rather than indexing, so it must be configured carefully to avoid blocking important pages.
Any site, especially large ones that need to steer crawl budget away from low-value URLs.
Set it up at launch, and review it during audits and after structural changes.
Our team can put this into practice for your site. Start with a free audit, or explore the technical seo service.
Crawlability is how easily a search engine's bots can discover and access a site's pages by following links and directives such as robots.txt.
IndexabilityIndexability is whether a crawled page is eligible to be stored in a search engine's index, governed by signals like noindex tags, canonicals, and content quality.
Technical SEOTechnical SEO is the practice of making a website easy for search engines to crawl, render, and index. It covers site architecture, speed, structured data, and indexation controls.
Last updated: 2026-08-03