SEO
Robots.txt Tester
Test whether specific URLs or paths are allowed or disallowed by pasted robots.txt rules directly in your browser.
Tool
Test whether specific URLs or paths are allowed or disallowed by pasted robots.txt rules directly in your browser.
This tool runs in your browser. Your input is processed locally and is not uploaded.
About this Tool
Paste robots.txt content, enter a user-agent and test one or more URLs or paths. The tester shows whether each path appears allowed or disallowed based on common robots.txt matching behavior and explains which rule matched.
Examples
- Test /admin/settings against User-agent: *
- Check if /products/item-123 is crawlable
- Compare Googlebot and wildcard rules
Frequently Asked Questions
What does the tester compare?
It compares your test URLs or paths against the pasted robots.txt rules for the selected user-agent.
Does it request the tested URLs?
No. It only evaluates the path text against robots.txt rules. No URLs are fetched.
Can I test multiple URLs at once?
Yes. You can paste multiple URLs or paths and review the matching result for each one.