Robots.txt Explained: How to Control What Search Engines Crawl
A robots.txt file is a set of instructions, not a lock — here's what it actually does, what it can't do, and how to write one that doesn't accidentally block your whole site.
TCTechToolsCenter Teamrobots.txt is a plain-text file at your site's root (yoursite.com/robots.txt) that tells well-behaved search engine crawlers which parts of your site they're allowed to crawl. It's a set of polite instructions, not a security mechanism — it doesn't password-protect anything, and a page can still get indexed if something else links to it, even if robots.txt disallows crawling it.
What robots.txt actually controls
- Crawling, not indexing — disallowing a URL stops crawlers from fetching its content, but the URL can still appear in search results (usually with no description) if it's linked elsewhere.
- Well-behaved bots only — legitimate search engines respect it; it does nothing to stop a scraper or bad actor that ignores the file entirely.
- Crawl budget — for large sites, blocking low-value sections (admin pages, search result pages, duplicate filtered views) helps crawlers spend more time on pages that matter.
Sponsored
The basic syntax
- `User-agent: *` — applies the following rules to all crawlers (or name a specific one, like `Googlebot`).
- `Disallow: /path/` — blocks crawling of that path and everything under it.
- `Allow: /path/exception` — carves out an exception within a disallowed path.
- `Sitemap: https://yoursite.com/sitemap.xml` — points crawlers to your sitemap, which genuinely helps discovery.
Step-by-step: create a robots.txt
- Open the Robots.txt Generator.
- List the paths you want to block (e.g. `/admin/`, `/search?`) — leave public content unblocked by default.
- Add your sitemap URL so crawlers can find it easily.
- Copy the generated file and upload it as `robots.txt` at your site's root, not in a subfolder.
Common mistakes
- Using robots.txt to try to keep a page out of Google entirely — if you need that, use a `noindex` meta tag instead; robots.txt only stops crawling, not indexing.
- Blocking CSS/JS files that Googlebot needs to render your page properly, which can hurt how well Google understands your page's content and layout.
- Forgetting robots.txt must live at the exact root (`/robots.txt`) — a copy in a subdirectory is ignored.
Tools used in this article
Sponsored
Frequently asked questions
Not reliably — it stops crawling, but a blocked page can still appear in search results (usually without a description) if something else links to it. Use a `noindex` meta tag if you need a page fully out of search results.
TechToolsCenter Team
Product & Tools
The team behind TechToolsCenter — building fast, private, browser-based tools and writing practical guides on how to get the most out of them.
Related articles
What Is a Canonical Tag, and Why Does It Matter for SEO?
The same page is almost always reachable through more than one URL — with or without www, with a tracking parameter, with a trailing slash. A canonical tag is how you tell Google which one actually counts.
How to Generate a Sitemap.xml for Your Website (Free)
A sitemap doesn't guarantee rankings, but it does help search engines find your pages faster — here's what actually belongs in one, and how to build it without writing XML by hand.
Core Web Vitals Explained: LCP, INP and CLS in Plain English
Three precise, measurable metrics Google uses to judge if a page actually feels fast and stable — here's what each one measures, its threshold for 'good,' and practical fixes for each.