Robots.txt Generator screenshot

Robots.txt Generator

Generate a standard robots.txt with crawl rules for your domain.

Privacy First
0 Runs
Updated 2026-11-07
Runs in your browser
No upload
No account needed
Nothing is fetched

Generate a standard robots.txt with crawl rules for your domain. Enter your numbers and get an instant result.

Fill in a sitemap address, a comma-separated list of paths to disallow and a list to allow, choose a crawl delay, and the page writes a robots.txt file you can copy or download.

The output is one group for every crawler: a User-agent line of an asterisk, an optional Crawl-delay, the Allow lines, the Disallow lines and the Sitemap line when an address is present. That order is the one the file is written in, and the preview updates on every keystroke.

Paths are normalised to start with a slash, so admin and /admin produce the same line. The page does not check that a path exists, that an allow rule overrides a longer disallow rule, or that the sitemap address resolves; it writes what you typed.

What the file can and cannot say

A robots.txt file is a request to crawlers, not a lock. A crawler that ignores it, a scraper, or a page reached by a direct link is not stopped by a Disallow line, and the file is public by definition because a crawler has to fetch it.

This page writes a single group. Per-crawler groups, wildcard and end-of-path patterns, and a Host directive for Yandex are all valid robots.txt features, but none of them is on this form, so the output is a plain file rather than a complete policy.

  • One User-agent group for all crawlers
  • Crawl-delay: none, 1, 2, 5 or 10 seconds
  • Sitemap line written only when the field is not empty

Order and precedence

The page emits every Allow line before every Disallow line. That is a readable convention, not a precedence rule: a crawler chooses the most specific matching rule, and when an allow and a disallow are equally specific the allow wins in the widely used interpretation.

Nothing is validated. A Disallow line without a leading slash is corrected for you, but a line containing a space, a scheme or a full URL is written as typed and will not do what you expect.

How to use the four fields

Type the sitemap address into Sitemap URL and the paths into the Disallowed and Allowed boxes as comma-separated lists; the File Output preview rewrites on every keystroke. Pick a crawl delay of none, 1, 2, 5 or 10 seconds from the dropdown when a slower crawl is wanted.

Press Download to save the file, or copy the preview text directly into the site root as robots.txt. Check the preview's Allow and Disallow lines against what was typed before publishing, since the page writes paths it does not verify.

What to Expect

Best for

  • Writing a small robots.txt for a site with a sitemap
  • Keeping a staging path out of a crawl
  • Producing a file to download and edit further

Not ideal for

  • Restricting access, which needs authentication rather than a text file
  • Per-crawler rules for Googlebot, Bingbot and others
  • Wildcard or end-anchor patterns the form does not offer

What this tool keeps

  • The path order you typed within the allow list and the disallow list
  • A leading slash on every path, added when missing
  • The exact sitemap address, untrimmed beyond its outer spaces

What may need cleanup

  • Path precedence, which a longer disallow can override unexpectedly
  • The sitemap address, which the page does not fetch or check
  • Per-crawler groups, which must be added by hand

Common errors

  • Expecting a Disallow line to hide a page from search results entirely
  • Putting a full URL in a Disallow field instead of a path
  • Leaving the sitemap field with an address that does not resolve

Example use cases

  • A small site publishing its first robots.txt
  • Blocking an admin path before a launch
  • A starter file for a developer to extend

Sample input

The sitemap https://example.com/sitemap.xml, disallow /admin, /private, /api, allow /assets, /static and no crawl delay.

Sample output

A User-agent asterisk group with two Allow lines, three Disallow lines and one Sitemap line, ready to copy or download.

Who this is for

  • Site owners adding a robots.txt
  • Developers preparing a launch
  • SEO beginners learning the file's shape

Frequently asked questions

What does the robots.txt generator write?

A sitemap line, disallow and allow paths, and a crawl delay, assembled into a robots.txt file you can copy or download.

Can robots.txt restrict access?

No. It is a crawler instruction, not authentication, so access control needs a real auth layer.

Does it support per-crawler rules or wildcards?

No. The form writes one group and does not offer per-crawler groups or wildcard patterns.

Developer program

Build custom tools for your store

Join the SDK program to create useful ecommerce utilities for your store.