Skip to content
Sumly

Robots.txt Generator

Block the right bots, expose the right pages, ship the file.

In

Out

User-agent: *
Disallow: /admin/
Disallow: /cart
Disallow: /checkout
Disallow: /thank-you

Sitemap: https://yoursite.com/sitemap.xml
Upload as robots.txt at your domain root. Disallow hides pages from crawling but not from indexing — use a noindex meta tag to keep pages out of results.

How Robots.txt Generator works

A robots.txt is a list of user-agent groups: each bot reads the first group that matches its name (or the * wildcard) and follows the Disallow/Allow rules inside it.

Three presets cover the common cases: standard (block private areas like /admin/ and checkout), allow-all (the public default) and block-all (staging, where you want nothing indexed).

The Sitemap line at the bottom is technically outside the standard but supported by every major engine — it tells crawlers where the full URL inventory lives. Crawl-delay is respected by Bing and most minor bots; Google ignores it (use Search Console settings instead).

Worked example

The standard preset with four disallowed paths and a sitemap URL produces a 8-line file: block /admin/, /cart, /checkout and /thank-you for all bots, then declare the sitemap. Upload it to https://yoursite.com/robots.txt — the root, or it does nothing.

The tool above opens with these numbers. Change them to run your own scenario, or press Reset to come back to this example.

Frequently asked questions

Paste URLs, get a valid sitemap.xml Google will accept first try.

Rich-result-ready JSON-LD for articles, products, FAQs and local.

Clean, keyword-dense slugs from any batch of titles.

See your result exactly as Google will render it — before you ship.

Six keyword-first titles, sized to Google's 60-character budget.

Benefit-led descriptions that fit 155 characters and earn the click.

  • robots.txt
  • crawl
  • technical seo
  • search console
  • crawler