Robots.txt Generator

Create a correct robots.txt with rules per crawler, crawl-delay, sitemap links and a one-click preset to block AI crawlers.

How to use Robots.txt Generator

1. Add a group for each user-agent you want to control.
2. Enter Allow and Disallow paths and your sitemap URL.
3. Copy or download robots.txt and upload it to your site root.

Frequently asked questions

How do I block AI crawlers like GPTBot?

Click "Preset: block AI crawlers" to add Disallow rules for the major AI bots.

Where do I put robots.txt?

In the root of your domain, for example https://example.com/robots.txt.