Robots.txt Builder screenshot

Robots.txt Builder

Build a robots.txt from a user-agent, allow and disallow paths and a sitemap URL, then copy or download it.

Privacy First
0 Runs
Updated 2026-11-07
Runs in your browser
No upload
No account needed
Nothing is fetched

A robots.txt assembled from its fields

A robots.txt file is a list of field lines: a user-agent group, the paths that group may or may not crawl, and an optional sitemap address. This page writes that file from four inputs and shows the result as you type.

The fields

The user-agent field names the crawler the group applies to; * means every crawler. Allow and disallow each take one path per line, and a line that does not start with a slash gets one, because a robots path is always absolute. The sitemap field takes one address and is emitted only when it is an absolute http or https URL, so a bare host is refused with a note rather than written as a line a crawler cannot use.

The output

The page writes User-agent, then the allow lines, then the disallow lines, then a blank line and the sitemap lines. That is the order crawlers expect, and every non-blank line is a field name, a colon and a value. Copy the text or download it as robots.txt and place it at the site root.

What it does not do

It does not fetch the site, test whether a crawler honours the rules, or verify that the sitemap exists. It does not replace a sitemap either: robots.txt points at one, and the two files do different jobs. Whether a path should be disallowed is an editorial decision this page cannot make for you.

A robots.txt file is a list of field lines: a user-agent group, the paths that group may or may not crawl, and an optional sitemap address. This page writes that file from four inputs and shows the result as you type.

The user-agent field names the crawler the group applies to, with an asterisk meaning every crawler. Allow and disallow take one path per line, and a line without a leading slash gets one, because a robots path is always absolute.

The sitemap field is emitted only when it is an absolute http or https address; a bare host is refused with a note rather than written as a line a crawler cannot use.

The output order

The file is written as User-agent, then the allow lines, then the disallow lines, then a blank line and the sitemap lines. That is the order crawlers expect, and every non-blank line is a field name, a colon and a value.

Copy puts the text on the clipboard and Download writes robots.txt. Place the file at the site root; a robots file anywhere else is not read by a crawler.

  • User-agent, allow, disallow and sitemap fields
  • Paths normalised to a leading slash
  • Copy or download robots.txt

What the file cannot do

It does not fetch the site, test whether a crawler obeys the rules or verify that the sitemap exists. It also does not replace a sitemap: robots.txt points at one, and the two files do different jobs.

Whether a path should be disallowed is an editorial decision. A robots file is a request, not access control, and a page reachable by a direct link is still reachable.

How to use the four inputs

Name the crawler in User-agent, leaving the asterisk when the group is for every crawler, then list the allowed and disallowed paths one per line. Enter the sitemap as an absolute http or https address; a bare host is refused with a note instead of being written as a broken line.

Copy the output or Download it as robots.txt, then place the file at the site root where crawlers read it. Read the emitted order of fields once before publishing, since allow lines always precede disallow lines by this page's convention.

What to Expect

Best for

  • Writing a small robots.txt from its fields
  • Pointing crawlers at a sitemap
  • Blocking an admin or staging path

Not ideal for

  • Restricting access, which needs authentication
  • Testing how a specific crawler interprets a path
  • A sitemap, which this page does not generate

What this tool keeps

  • The user-agent value, defaulting to an asterisk
  • One path per line with a leading slash
  • A sitemap line only for an absolute http(s) address

What may need cleanup

  • Path precedence, which a crawler resolves rather than this page
  • The sitemap address, which is not fetched or checked
  • Per-crawler groups, which need a second file or a hand edit

Common errors

  • Entering a bare host in the sitemap field and finding no line
  • Expecting a disallow to make a page private
  • Placing the downloaded file somewhere other than the site root

Example use cases

  • A first robots.txt for a small site
  • Blocking a staging path before launch
  • Pointing at an existing sitemap

Sample input

A user-agent of asterisk, allow /assets and /static, disallow /admin and /api, and the sitemap https://example.com/sitemap.xml.

Sample output

A robots.txt with the user-agent line, two allow lines, two disallow lines, a blank line and the sitemap line, ready to copy or download.

Who this is for

  • Site owners adding a robots.txt
  • Developers preparing a launch
  • SEO beginners learning the field lines

Frequently asked questions

What does the robots.txt builder write?

A user-agent group with allow and disallow paths and an optional sitemap line, from four inputs, shown as you type.

Can it restrict access?

No. A robots.txt is a crawler instruction; access control needs authentication.

Why can a longer disallow override a shorter one?

Path precedence is resolved by the crawler, so the order and length of paths can surprise; test the file against the crawler you care about.

Developer program

Build custom tools for your store

Join the SDK program to create useful ecommerce utilities for your store.