To
ols
Tools
WEBTOP
Toggle navigation
Robots.txt Generator
Home
Robots.txt Generator
Sponsored
Robots.txt Generator
Super simple robots.txt Generator
General Options
Default: All robotos are:
Allowed
Not Allowed
Crawl Delay
Default - No Delay
5 Seconds
10 Seconds
20 Seconds
60 seconds
120 Seconds
Sitemap
Allow / Disallow specific Bots
Google
Same as Default
Allowed
Refused
Google Image
Same as Default
Allowed
Refused
Google Mobile
Same as Default
Allowed
Refused
MSN Search
Same as Default
Allowed
Refused
Yahoo
Same as Default
Allowed
Refused
Ask/Teoma
Same as Default
Allowed
Refused
Cuil
Same as Default
Allowed
Refused
GigaBlast
Same as Default
Allowed
Refused
Scrub The Web
Same as Default
Allowed
Refused
DMOZ Checker
Same as Default
Allowed
Refused
Nutch
Same as Default
Allowed
Refused
Alexa/Wayback
Same as Default
Allowed
Refused
Baidu
Same as Default
Allowed
Refused
Yahoo MM
Same as Default
Allowed
Refused
Disallow specific items/directories
Restricted items
Clear
Submit
Comments
Please enable JavaScript to view the
comments powered by Disqus.