Robots.txt Generator
Configure crawl rules for search engine bots and instantly generate a standards-compliant robots.txt file. Lightweight server processing — no page reloads.
Use * for all crawlers.
One path per line. Blocks crawlers from these URLs.
Optional overrides for blocked sections.
How to Use Robots.txt
Set your target User-agent (* applies rules to all bots).
Add Disallow paths for sections you want hidden from search engines (e.g. admin panels).
Optionally add Allow paths to unblock specific URLs inside a blocked area.
Click Generate, then upload the file to your site root as https://yoursite.com/robots.txt.
Frequently Asked Questions
Where do I upload robots.txt?
Place it in your website's root directory so it is accessible at yoursite.com/robots.txt. Most hosting panels and CMS platforms have a file manager for this.
Does robots.txt guarantee pages won't be indexed?
No. It is a polite request to crawlers. For stronger control, use noindex meta tags or password-protect sensitive pages.
What is Crawl-delay?
It asks bots to wait a number of seconds between requests. Support varies — Bing respects it; Google largely ignores it in favor of Search Console settings.
Should I include a Sitemap line?
Yes, when you have a sitemap. Adding Sitemap: https://example.com/sitemap.xml helps search engines discover your URLs faster.