Robots.txt Generator
Block the right bots, expose the right pages, ship the file.
In
Out
User-agent: * Disallow: /admin/ Disallow: /cart Disallow: /checkout Disallow: /thank-you Sitemap: https://yoursite.com/sitemap.xml
How Robots.txt Generator works
A robots.txt is a list of user-agent groups: each bot reads the first group that matches its name (or the * wildcard) and follows the Disallow/Allow rules inside it.
Three presets cover the common cases: standard (block private areas like /admin/ and checkout), allow-all (the public default) and block-all (staging, where you want nothing indexed).
The Sitemap line at the bottom is technically outside the standard but supported by every major engine — it tells crawlers where the full URL inventory lives. Crawl-delay is respected by Bing and most minor bots; Google ignores it (use Search Console settings instead).
Worked example
The standard preset with four disallowed paths and a sitemap URL produces a 8-line file: block /admin/, /cart, /checkout and /thank-you for all bots, then declare the sitemap. Upload it to https://yoursite.com/robots.txt — the root, or it does nothing.
The tool above opens with these numbers. Change them to run your own scenario, or press Reset to come back to this example.
Frequently asked questions
Related tools
Paste URLs, get a valid sitemap.xml Google will accept first try.
Rich-result-ready JSON-LD for articles, products, FAQs and local.
Clean, keyword-dense slugs from any batch of titles.
See your result exactly as Google will render it — before you ship.
Six keyword-first titles, sized to Google's 60-character budget.
Benefit-led descriptions that fit 155 characters and earn the click.
- robots.txt
- crawl
- technical seo
- search console
- crawler