Ad Slot
Robots.txt Generator
Generate robots.txt files to control search engine crawling.
Ad Slot
Ad Slot
Frequently asked questions
Can I add multiple Disallow paths under the same user-agent?
Not within a single rule — each rule has just one Disallow and one Allow field. To block multiple paths for the same bot, add another rule with the same User-agent value; robots.txt allows repeating User-agent blocks like that.
Does it check that my paths start with a slash?
No — whatever you type into Disallow or Allow is used exactly as entered, so it's on you to make sure a path like /admin/ actually starts with a forward slash.
Can I list more than one sitemap?
Not through the form — there's a single Sitemap URL field, so only one Sitemap: line gets added. You'd need to add extra sitemap lines to the downloaded file yourself.
Ad Slot
Related tools
Ad Slot