./prontouso.com

SEO & WEB TOOLS

robots.txt Generator

Build, edit, and test a robots.txt file directly in your browser. Add crawler groups, rules, and sitemaps, then copy or download the finished file.

Tool statusRuns in your browser
Preview
prontouso://network/robots.txt
LOCALUTF-8AUTO UPDATE

Configuration

The rules below update the file automatically.

INPUT
Templates
Group 1
Choose a shortcut or write your own.
Allowed rules
1 rule
Allow
Blocked rules
0 rules
Sitemaps
0

No sitemaps added.

File and test

Review, copy, download, or manually adjust the final content.

OUTPUT
robots.txtAuto-generated
2 lines23 characters

Test a crawler's access

Simulate how the current file responds for a specific bot and path.

Access allowed
The Allow / rule is the most specific match.
Processed locally in your browser Changes applied automatically

How it works

  1. Enter your input

    Fill in the values, paste your text, or upload the file this tool works with.

  2. See results instantly

    Most tools update live as you type; a few use a single button. Either way, the result appears right on this page.

  3. Use your results

    Copy, download, or share what the tool produces — you're always in control of the output.

Privacy and processing — Runs locally in your browser. This tool does not upload your input.

What is robots.txt Generator?

The Robots.txt Generator is a tool for building Allow and Disallow rules across one or more user-agent groups, manually editing the final file, and testing whether a specific path is allowed or blocked for a bot.

UNDERSTAND THE TOOL

How to generate and test robots.txt

Add one or more crawler groups, build lists of allowed and blocked rules per group, add any number of sitemaps, and test whether a path is allowed for a bot — all on this page, with a manual editor and ready-made templates for when you need more control or a faster start.

Choosing user-agents and groups

Use the All, Googlebot, or Bingbot shortcuts to fill a group's User-agent field quickly, or type any other name — the shortcut switches to "Custom" automatically once the text no longer matches a preset. "Add user-agent" adds a second (or third) bot to the same group when they should share identical rules. "Add group" creates an independent group instead, for a bot that needs different rules from everyone else. A User-agent: * group applies to any bot that doesn't have its own more specific group — Google explicitly does not combine a named bot's group with the * group, so a specific group needs to repeat any rule it should inherit.

Building the allowed and blocked rules

Each group's "Allowed rules" and "Blocked rules" lists accept as many paths as you need — use "Add rule" to create a new line and the X next to a field to remove it. Every change updates the file below instantly. The "Sitemaps" list at the bottom works the same way and accepts zero, one, or several sitemap URLs.

Templates

The template row above the groups fills in a common starting point in one click — allow everything, block everything, block one folder, allow a page inside a blocked folder, a Googlebot-specific setup, an empty file, or the illustrative example with an admin/private block and a sitemap. Applying a template replaces the current groups and sitemaps, so use Restore or undo in the editor if you want the previous version back.

Auto-generated file, manual editing, and Restore

While you use the form, the file shown in the editor carries an "Auto-generated" badge. Editing the text directly in the editor switches that badge to "Manually edited" and decouples the content from the form, so you can paste or adjust a fuller file with multiple groups. The Restore button discards the manual edit and rebuilds the file from the form's current groups and sitemaps.

Wildcards: * and $

A * in a path matches any run of characters, so Disallow: /*.pdf blocks every PDF regardless of folder. A trailing $ anchors the end of the URL, so Disallow: /*.pdf$ blocks /file.pdf but allows /file.pdf?preview=1, since the query string means the URL doesn't actually end in .pdf.

How the path test decides

The tester reads the current file content, finds the group whose user-agent most specifically matches the bot you entered (falling back to the * group only when no named group matches), and applies the most specific — meaning longest — matching Allow or Disallow rule for the tested path. If rule lengths tie, Allow wins, matching Google's own documented behavior of using the least restrictive rule on a conflict.

Before publishing

  • Test important paths, such as /, /admin/, /blog/post, and image files, for every bot that has its own group.
  • Keep sensitive rules simple and explicit, and add every Sitemap line you have.
  • After publishing at /robots.txt, validate it in the official tools of the search engines that matter for your site.

Privacy and robots.txt limits

The rules, the editor, and the tester run entirely in your browser — nothing is sent to a server. But robots.txt is a public crawling hint, not a security layer: anyone can read the published file, so never use it to hide private or admin URLs. It also only controls crawling, not indexing — a blocked page can still appear in search results if other sites link to it, since Google never gets to see a noindex tag on a page it isn't allowed to crawl.

Frequently Asked Questions

Does Disallow hide a page from everyone?

No. robots.txt only gives crawler instructions. It does not block users, secure private URLs, or remove pages that are already indexed — a disallowed URL can still show up in search results if it's discovered through an external link.

Which rule wins if Allow and Disallow both match?

The longest matching path wins. If both matches have the same length, Allow wins — this mirrors Google's documented behavior of applying the least restrictive rule when rules conflict.

What do * and $ mean in a path?

* matches any run of characters, so /*.pdf matches a .pdf file anywhere in the path. A trailing $ anchors the end of the URL, so /*.pdf$ only matches a path that ends exactly in .pdf, not one followed by a query string like ?id=1.

Do I need a separate group for Googlebot or Bingbot?

Only when those bots need different instructions from everyone else. Use the Googlebot or Bingbot shortcuts to fill a group's user-agent quickly; without a specific group, User-agent: * covers every compliant bot. A named group's rules replace the * group's rules for that bot entirely — they aren't combined.

Can a robots.txt file have more than one Sitemap line?

Yes. A file can list zero, one, or several Sitemap URLs; use "Add sitemap" to add more.

Where should I put the file?

Publish it at the root of the site, such as https://example.com/robots.txt.

What does the "Manually edited" badge mean, and what does Restore do?

As soon as you edit the file's text directly in the editor, it stops reflecting the form and shows a "Manually edited" badge instead — handy for pasting in a ready-made file with several groups. The Restore button discards that edit and rebuilds the file from the form's current groups and sitemaps.

Can I add as many groups, user-agents, rules, and sitemaps as I want?

Yes. "Add group" creates another crawler group, "Add user-agent" adds a bot to the current group, and "Add rule" and "Add sitemap" work the same way on their own lists — each with an X to remove that specific entry.

Related Tools