Robots.txt Generator
Set which crawlers can reach which paths and copy a ready-to-upload robots.txt file.
How to use Robots.txt Generator
- Set a user-agent, then add Disallow and Allow paths for it — use "+ Add rule" for more than one user-agent.
- Add your sitemap URL if you have one.
- Copy the generated robots.txt and upload it to your site's root.
Frequently asked questions
Can I add multiple Disallow paths under the same user-agent?
Not within a single rule — each rule has just one Disallow and one Allow field. To block multiple paths for the same bot, add another rule with the same User-agent value; robots.txt allows repeating User-agent blocks like that.
Does it check that my paths start with a slash?
No — whatever you type into Disallow or Allow is used exactly as entered, so it's on you to make sure a path like /admin/ actually starts with a forward slash.
Can I list more than one sitemap?
Not through the form — there's a single Sitemap URL field, so only one Sitemap: line gets added. You'd need to add extra sitemap lines to the downloaded file yourself.
Community
Loading...