Match rules to policy
Product, legal, and SEO teams often disagree on what should be discoverable. Use this builder to prototype directives, then run the result past stakeholders before it becomes your public contract with crawlers.
Layer with meta robots
Disallow keeps URLs out of diligent crawlers’ habitual fetches—it does not guarantee removal from indexes that already memorized URLs. Combine with canonical tags and robots meta or HTTP headers where you must suppress snippets.
Frequently asked questions
Does robots.txt block scrapers completely?
No—treat it as guidance for cooperative bots, not enforcement against aggressive ones.
Where do I upload robots.txt?
At the site root path `/robots.txt` on the public hostname users and crawlers resolve.
Does Google honor Crawl-delay?
Generally no—plan capacity and Search Console tooling instead of relying on that directive for Googlebot.
Multiple sitemaps?
List each Sitemap: URL on its own line; splitting feeds by vertical is common.
Is configuration uploaded?
No—assembly stays offline in the tab.