How this tool works
Combines the user-agent, allow, disallow, and optional sitemap values you enter into plain-text directives.
How to use it
- Choose the crawler user-agent scope.
- Add only paths you intentionally want to control.
- Review with a robots tester and publish at the correct host root.
Limits to keep in mind
Rules apply by path and crawler behavior. Robots.txt does not provide access control and a blocked URL may still be indexed without its contents.
Frequently asked questions
Does Disallow remove a URL from Google?
No. Crawling controls are different from indexing controls; use an appropriate noindex or access restriction when needed.
Can I block my whole site accidentally?
Yes. Review every rule carefully before publishing.
Where does the robots.txt file go?
At the root of the host, for example https://example.com/robots.txt. Rules in a file placed in a subfolder are not read, and each subdomain needs its own file.
Should I block CSS and JavaScript files?
Generally no. Search engines may need them to render and understand your pages, so block only paths you have a clear reason to keep crawlers away from.
What a robots.txt file does
A robots.txt file is a plain text file placed at the root of a host that tells compliant crawlers which paths they may or may not request. It is part of the Robots Exclusion Protocol, and it works by crawler cooperation, not by enforcement. Well-behaved search engine crawlers read it before crawling and follow it. Malicious bots can ignore it completely.
The Robots.txt Generator combines the values you enter into the standard directives. It writes a User-agent line to say which crawler the group applies to, Allow and Disallow lines for paths, and an optional Sitemap line that points to your XML sitemap. It prints the result as plain text that you can copy into a file named robots.txt. The tool does not upload anything or edit your website. It only builds the text, and you decide whether it is right for your site.
The directives you will use most
User-agent names the crawler a group of rules applies to. An asterisk means every crawler that has no more specific group. Disallow lists a path prefix that the crawler should not request. Allow lists a path that may be requested, which is mainly useful for opening a subfolder inside a disallowed folder. An empty Disallow line means nothing is blocked for that group.
Google supports two pattern characters in paths: an asterisk matches any sequence of characters, and a dollar sign marks the end of a URL. Paths are case sensitive, so /Photos/ and /photos/ are different. When more than one rule matches a URL, Google uses the most specific rule, which is the one with the longest matching path. The Sitemap line must contain a full absolute URL and can appear anywhere in the file. Crawl-delay is not supported by Google, although some other crawlers honor it.
Crawling control is not indexing control
The most common robots.txt mistake is using it to hide pages from search results. A disallow rule stops a compliant crawler from reading the page, but it does not stop the URL from being indexed. If other pages link to a blocked URL, a search engine can still list it with no description, because it knows the URL exists but cannot see the content.
To keep a page out of search results, let it be crawled and use a noindex robots meta tag or an X-Robots-Tag header, or put the page behind a login. Blocking a page in robots.txt can even prevent a noindex tag from being seen, which defeats the purpose. Robots.txt is also not a privacy tool: the file itself is public, so listing a secret folder in it advertises the folder to anyone who looks.
Decide which paths deserve a rule
Start with the smallest possible file. Most small sites need only a user-agent line, an empty disallow, and a sitemap line. Add rules only when you have a specific reason, for example internal search result pages that create endless URL variations, cart and checkout steps, admin areas, or staging folders that should not use up crawl time.
Avoid blocking CSS, JavaScript and image files that your pages need to render. Search engines use them to understand layout and mobile friendliness, and blocking them can lower the quality of what is understood about your page. Be careful with broad patterns. A rule such as Disallow: /p will block every path that starts with /p, including /products and /pricing. Write each path with the exact folder or file you mean, and include the trailing slash when you mean a folder.
How to use the generator step by step
First choose which crawler the group applies to. Use the all-crawlers option unless you have a reason to write separate rules for a specific bot. Next add the paths you want to control, one per line, starting each with a forward slash. Add your sitemap URL as a full address including https. Then review the generated lines and read them aloud as sentences, such as "for all crawlers, do not request the admin folder".
Copy the text into a file named exactly robots.txt, using UTF-8 plain text without formatting. Upload it to the root of the host, so it is reachable at https://yourdomain.com/robots.txt. The file applies only to the exact protocol and host it is served from, so https://www.example.com and https://blog.example.com each need their own file. A robots.txt in a subfolder is ignored.
Test before and after you publish
Always test a new robots.txt before relying on it. Google Search Console includes a robots.txt report that shows the version Google last fetched and any errors it found, and the URL Inspection tool tells you whether a specific URL is blocked. Open the file in your browser to confirm it loads with a successful status and displays plain text.
Be aware of how status codes are treated. If the file returns a not found error, crawlers generally act as though nothing is restricted. If it returns a server error for an extended time, Google may slow or pause crawling of the site, so make sure the file is reliably available. Google also reads only a limited amount of the file, around 500 kibibytes, so keep it short. After publishing, give crawlers time to fetch the new version, since caching can delay changes by up to a day.
Common mistakes and limits of this tool
The tool cannot see your website. It does not know which URLs exist, which paths matter, or whether a rule will block something important. It cannot test how a crawler will interpret your rules, and it does not verify the sitemap URL you enter. Treat the output as a draft that you review.
The mistakes that cause the most damage are blocking the entire site with Disallow: / left over from a staging environment, blocking resource folders, using robots.txt as a security measure, and putting noindex lines in the file, which Google no longer supports. Another is forgetting that each subdomain has its own file. Keep a copy of the previous version before you change anything, and recheck the file after every site migration or platform update.
More SEO Tools · Browse the tools catalogue