Robots.txt Generator
Generate robots.txt files for your website with common presets
Block AI Crawlers
Check to add Disallow: / rules for AI bots
Generated robots.txt
URL Tester
Check whether a URL path would be blocked or allowed by your current rules
Was this tool helpful?
What is Robots.txt Generator?
Free online robots.txt generator to generate a properly formatted robots.txt file for your website in seconds.
Select which search engine crawlers to allow or block, specify disallowed paths, set a crawl delay, and add your sitemap URL. Block all bots, allow only Googlebot, or add a rule to block AI crawlers like GPTBot and CCBot. The tool includes presets for WordPress, Shopify, and other platforms, plus a way to disallow a specific folder while allowing the rest of the site. Validates your rules for correctness. Essential for SEO and controlling how search engines index your site. No signup required - create and download your robots.txt file instantly.
How to Use Robots.txt Generator
- 1Select which crawlers to configure rules for (all crawlers, Googlebot, Bingbot, etc.)
- 2Add allowed and disallowed paths for each crawler group
- 3Optionally set a crawl delay and add your sitemap URL
- 4Copy the generated robots.txt content or download it as a file
Frequently Asked Questions
What is a robots.txt file?
Does robots.txt block pages from appearing in search results?
Should I include my sitemap URL in robots.txt?
What crawlers should I configure rules for?
How do I block all bots from my site?
How do I stop AI crawlers like GPTBot from scraping my content?
How do I disallow one folder but allow the rest of the site?
Related Tools
Multilingual seo
Generate hreflang tags for multilingual SEO - HTML, HTTP headers, and XML sitemap formats
Sitemap generator online
Free online XML sitemap generator - generate XML sitemaps from a list of URLs
Rich results
Generate Schema.org JSON-LD structured data for SEO - 17 schema types with validation
Create package.json
Generate package.json files with a form-based builder - 8 project presets, all fields supported
Htaccess rewrite rules
Free online htaccess generator - generate Apache .htaccess redirect and rewrite rules
Meta Tag Generator
Free online meta tag generator - generate SEO meta tags, Open Graph, and Twitter Card markup for your website
Env editor
Edit and validate .env files online - detect duplicates, mask secrets, generate .env.example
.gitignore Generator
Generate .gitignore files for any project - 50+ templates for languages, frameworks, IDEs, and OS
Keep reading: JSON Syntax Cheat Sheet and file permissions reference. Doing this in code instead? See the tool chain endpoints.