Every site with a robots.txt file is making a request, not issuing a command. It's one of the oldest conventions on the web -- a plain text file that tells well-behaved crawlers which parts of a site they're welcome to visit. People frequently assume it does more than that, which causes real problems when a page they thought was hidden shows up in search results anyway.
What it actually does
A robots.txt file sits at the root of a domain (yoursite.com/robots.txt) and lists rules for user agents -- specific bots or all of them -- telling each which paths it may or may not crawl. A search engine that respects the standard will read this file before crawling and skip disallowed paths. That's the entire mechanism: a set of crawl instructions, checked voluntarily by the crawler.
Why it's not a privacy or security tool
The word "disallow" makes it sound like a barrier, but it isn't one. Three things follow from that:
- It only works if the bot chooses to obey it. Reputable search engines do. Plenty of scrapers, spam bots, and malicious actors simply ignore it -- there's no enforcement mechanism baked into the protocol.
- A disallowed URL can still get indexed. If other sites link to a page you've disallowed, a search engine can list that URL in results (often with no description, since it never crawled the content) purely because it knows the URL exists.
- The file itself is public. Anyone can view it at /robots.txt, which means listing a sensitive path there to "hide" it actually advertises that path to anyone reading the file.
If you need a page to genuinely stay out of search results, the correct tool is a noindex meta tag or HTTP header on that page, not a robots.txt rule. If you need it to stay private, put it behind authentication -- robots.txt was never designed for that job.
What it's actually good for
Despite the limits, it's still useful for its intended purpose: crawl management. Common legitimate uses include keeping crawlers out of admin paths, internal search result pages, staging environments, or duplicate content generated by filters and sorting parameters -- all cases where you don't care about secrecy, you just don't want bots wasting crawl budget or indexing low-value pages. It's also where you can point crawlers to your sitemap with a Sitemap: directive, which helps them discover your important pages faster in the first place.
Getting the syntax right
The format is simple but easy to get subtly wrong -- a missing User-agent line, a trailing slash that changes what a rule matches, or a Disallow: / left over from a staging site that quietly blocks an entire production domain from search engines. Hand-writing it works, but it's easy to introduce exactly that kind of mistake.
The Robots.txt Generator builds a correctly formatted file from simple inputs, and pairing it with the Sitemap Generator gives crawlers both pieces at once: what to skip, and where to find everything else.