Robots.txt Generator

Generate a valid robots.txt file to control how search engines crawl your website.

Quick Start Presets
Configuration
One path per line
One path per line
Delay between requests (not supported by Google)
Preferred domain for Yandex. Most search engines ignore this directive.
Add notes at the top of your robots.txt file
How to Use
  1. Select a preset or start with custom configuration
  2. Add user-agent rules for specific crawlers
  3. Define Allow and Disallow paths
  4. Add your sitemap URL(s)
  5. Generate and download your robots.txt
  6. Upload to your website's root directory
Your robots.txt file should be accessible at https://yourdomain.com/robots.txt
Directive Reference
User-agent
Specifies which crawler the rules apply to. Use * for all crawlers.
Disallow
Tells crawlers not to access specific paths. Empty value means allow all.
Allow
Explicitly allows crawling of specific paths (overrides Disallow).
Sitemap
Points to your XML sitemap location. Multiple sitemaps are allowed.
Crawl-delay
Seconds between requests. Note: Google ignores this directive.
Host
Preferred domain (mainly for Yandex).
AI Crawlers

Many AI companies now respect robots.txt. Common AI crawlers include:

  • GPTBot - OpenAI
  • Google-Extended - Google AI
  • CCBot - Common Crawl
  • anthropic-ai - Anthropic

Block these if you don't want your content used for AI training.

Related Tools