Robots.txt Generator
Build a robots.txt file — set the user-agent, disallow paths and point crawlers to your sitemap.
Build a robots.txt file that tells search engine crawlers which parts of your site they may and may not visit. Choose which crawler the rules apply to, list the paths you want to keep out of, and add a link to your sitemap — then copy the file for your site's root. It's a small file with an outsized role in how your site is crawled. Everything runs in your browser.
How robots.txt works
The file lives at the root of your domain (example.com/robots.txt) and is the first thing well-behaved crawlers check. Each block names a user-agent (a specific crawler, or * for all) and lists Disallow rules — the path prefixes that crawler should skip. You can also point to your sitemap so crawlers find your full URL list. This tool assembles those pieces into correct syntax so a stray typo doesn't accidentally block your whole site.
What it can and cannot do
An important distinction: Disallow asks crawlers not to crawl a path — it does not guarantee a page stays out of search results, and it does not protect private content. Reputable crawlers obey it; malicious bots ignore it entirely. To keep a page out of the index, use a noindex meta tag instead, and to protect sensitive content, use real access controls like passwords. Think of robots.txt as crawl-management guidance for cooperative bots, not as a security or privacy tool.
Frequently Asked Questions
Where does the robots.txt file go?
At the root of your domain, reachable at example.com/robots.txt. It won't work in a subfolder — crawlers only look for it at the root.
Does disallowing a page remove it from Google?
No. Disallow asks crawlers not to crawl the page, but a disallowed URL can still appear in results if it's linked elsewhere. To keep a page out of the index, use a noindex meta tag instead.
Can robots.txt protect private or sensitive pages?
No. It's only a request that cooperative crawlers respect; malicious bots ignore it, and the file itself is publicly readable. Protect sensitive content with real access controls like authentication.
Should I add my sitemap to robots.txt?
Yes, it's good practice. A Sitemap line pointing to your sitemap.xml helps crawlers discover all your URLs and is independent of the allow/disallow rules.