SEO Tools

Robots.txt
Generator

Build a valid robots.txt to tell search engines what to crawl. Start from a preset, add disallow rules and your sitemap, then copy it to your site root.

Crawler rules
Sitemap line
Presets
User-agent: *
Disallow: /admin/
Allow: /

Sitemap: /sitemap.xml
robots.txt

            

Frequently Asked Questions

Where do I put robots.txt?
It must live at the root of your domain — https://example.com/robots.txt. Crawlers always look there; it will not work in a subfolder.
Does Disallow hide a page from Google?
No. Disallow stops crawling, but a blocked URL can still be indexed if other sites link to it. To keep a page out of search results, use a noindex meta tag and allow crawling.
What does User-agent: * mean?
The asterisk applies the rules to all crawlers. You can target a specific bot instead, e.g. User-agent: Googlebot, to give it different rules.
Should I add my sitemap to robots.txt?
Yes. A Sitemap: line pointing to your sitemap.xml helps search engines discover all your pages faster. It is independent of the allow/disallow rules.

Related Tools

Redirect Generator Nginx & Apache Config Word & Character Counter

Put Your Site on a Fast, SEO-Ready VPS

TextraHost Linux VPS plans start at $5.50/mo — full root, NVMe storage, India & Singapore.

Trusted & Certified
HostAdvice Silver — Top 25 Reseller Hosting HostAdvice — Top 25 Offshore Hosting 2026 HostAdvice — Over 99% Uptime 2025 Website Planet Feature Accredited .xyz Registrar tawk.to Partner Green hosting verified by The Green Web Foundation AbuseIPDB Contributor Badge TextraHost on PeerPush
Join Now
Join our Telegram channel Deals, updates & support on the go.
TextraHost AI replied