What robots.txt Actually Does (and Doesn't Do) for SEO

robots.txt is a plain-text file at a site's root that tells well-behaved crawlers which parts of a site they're allowed to visit. It's a simple, decades-old standard, but the single most common misunderstanding about it causes real, avoidable SEO mistakes.

Disallow controls crawling, not indexing

Blocking a page in robots.txt tells crawlers not to visit that page — it does not guarantee the page stays out of search results. If other pages link to a disallowed URL, search engines can still index that URL based on the link context alone, sometimes showing it in results with no description, since the crawler was never allowed in to read the actual content. To reliably keep a page out of search results, the correct tool is a noindex meta tag on the page itself, which requires crawlers to be allowed in to read that instruction.

What robots.txt is actually good for

It's the right tool for keeping crawlers away from pages that provide no SEO value and waste crawl budget — internal search result pages, admin areas, staging environments, or duplicate content generated by URL parameters. For a small site, crawl budget rarely matters in practice, but for a large site with thousands of pages, directing crawlers away from low-value URLs helps them spend more time on pages that matter.

The sitemap reference is worth including

Adding a Sitemap: line to robots.txt is a low-effort way to point crawlers directly to your sitemap.xml, giving them a clear map of every page you want indexed rather than relying entirely on them discovering pages through internal links.