robots.txt Generator
Build a robots.txt by listing paths to disallow, with user-agent targeting, block-all mode, and sitemap URL. Everything runs in your browser.
FAQ
Where does robots.txt go?
At the site root (https://example.com/robots.txt)—crawlers ignore it anywhere else.
Does robots.txt hide pages completely?
No—it's only a crawling request, not access control. Use authentication or noindex for real hiding.
Can I block AI crawlers?
Yes—generate separate blocks for GPTBot, CCBot, Google-Extended, etc., and concatenate them.
Related tools
Generate a table of contents with GitHub-style anchor links from your Markdown headings. Everything runs in your browser.
Percent-encode Korean and special characters for URLs, or decode them back.
Compute the network and broadcast addresses, host range, subnet mask, and usable host count from CIDR notation (e.g. 192.168.1.0/24). Everything runs in your browser.
Pick Linux file permissions with checkboxes to get the octal (755) and symbolic (rwxr-xr-x) forms, or set them from an octal value. Everything runs in your browser.
Parse a cron expression field by field and compute the next 5 run times.
Compare two texts line by line and highlight added and removed lines.