Robots.txt Generator
Create robots.txt files to control search engine crawling
Generate robots.txt files to manage how search engines crawl your website.
- Add rules for different user agents
- Specify allowed and disallowed paths
- Set crawl delays and sitemaps
- Preview and download robots.txt
- Common presets for popular bots
Advertisement
Add Rules
Added Rules
| User Agent | Rule Type | Path | Crawl Delay | Action |
|---|