WEBMASTER TOOLS
Robots.txt Tester & Generator
Test how robots.txt rules match a crawler and path, then generate a starter robots.txt file — locally in your browser.
LOCAL ROBOTS.TXT CHECK
Test a crawler against your rules
Supports User-agent, Allow, Disallow, Sitemap, path wildcards (*) and end anchors ($). Unsupported directives are reported as warnings.
Test result
- Crawler
- Path checked
- Matching rule
Review these warnings
Your pasted policy and test values stay in this browser. Nothing is uploaded.
ROBOTS.TXT GENERATOR
Build a starter policy
Enter one path or pattern per line. The generated file contains a single User-agent: * group.
Each path must begin with /. Use * as a wildcard and $ to match the end.
Copied to clipboard.
How to test and generate robots.txt rules
Paste a robots.txt policy, enter a crawler's user-agent and the URL or path you want to check, then select Test path. The result shows whether the most specific matching rule allows or disallows the request. Use the generator to prepare a simple wildcard policy and optional sitemap declaration.
How rule matching works
- The most specific matching user-agent group is selected; groups with the same matching agent are combined.
- Among matching Allow and Disallow rules, the rule with the longest matching path takes precedence. Allow wins when matching rules have equal length.
- An unmatched path is allowed by default. The
*wildcard matches any sequence;$anchors a pattern to the end of the path. - Malformed lines, directives before a user-agent group and unsupported directives are reported as warnings.
This is a local text checker, not a live crawler. It cannot verify which policy your server currently serves, follow redirects, or test how every search engine interprets extensions. Review RFC 9309 and the relevant crawler documentation before relying on a result.
Important security limitation
Robots.txt is not an access-control mechanism. Robots Exclusion Protocol rules request that compliant crawlers avoid selected URLs; they do not prevent people or other clients from accessing those URLs. Do not use robots.txt to protect confidential information. Apply authentication and server-side authorization, and avoid listing sensitive paths publicly in a robots file.
Search engines may apply their own rules and crawler-specific extensions, so verify behavior against the documentation for each crawler you care about.
Frequently asked questions
Does this tool fetch my live robots.txt file?
No. Paste robots.txt text or use the generator. The page never requests your site or sends your inputs to a server.
Does a robots.txt rule keep private pages secure?
No. Robots.txt is a crawl directive, not access control. Use authentication or server-side authorization to protect private content.
Why can a search engine behave differently from this result?
Robots.txt is a convention with crawler-specific details and extensions. This tool applies the matching rules described by RFC 9309; individual crawlers may have documented differences and can ignore directives they do not support.
What does the generator include?
It creates one wildcard User-agent group from your Allow and Disallow paths and can add an absolute HTTP or HTTPS Sitemap URL. Review the output before publishing it at your site root.
Sources
Privacy note: rules, paths and generated text are processed locally in your browser and are not sent to NAB Tools.