Generador de robots.txt
Crea un archivo robots.txt, edita las reglas finales, comprueba si una ruta está permitida para un user-agent y descarga el archivo.
Rules
robots.txt
Blocked · disallow /admin
How to generate and test robots.txt
Build a robots.txt file with Allow, Disallow, User-agent, and Sitemap lines, then test whether a bot can crawl a specific path.
Rules and groups
A robots.txt group starts with User-agent and then lists Allow or Disallow paths for that bot. The generator creates one group from the form, while the editable output lets you paste or adjust a fuller file.
Testing a path
The tester parses the current robots.txt text, finds user-agent groups that match the bot name, and applies the longest matching Allow or Disallow rule. If rule lengths tie, Allow wins.
Before publishing
- Test important public pages and blocked admin paths.
- Add a Sitemap line when you have a sitemap URL.
- Remember that robots.txt is a crawl hint, not access control for private content.
Publishing notes
A robots.txt file is public by design. Do not list secret URLs as a privacy measure; protect private areas with authentication instead.
Preguntas frecuentes
Does Disallow hide a page from everyone?
No. robots.txt only gives crawler instructions. It does not block users, secure private URLs, or remove pages that are already indexed.
Which rule wins if Allow and Disallow both match?
The longest matching path wins. If both matches have the same length, this tester treats Allow as the winner.
Can I test Googlebot separately from other bots?
Yes. Enter the bot name in the tester. A group for * also applies when no more specific matching group changes the outcome.
Where should I put the file?
Publish it at the root of the site, such as https://example.com/robots.txt.