robots.txt builder & checker

Build a robots.txt from rules, or check whether a path is allowed by pasted rules.

Text is processed in this browser — nothing is sent anywhere.

Google-style longest-match semantics: the most specific user-agent group applies, the longest path wins, and Allow wins ties. Crawl-delay and other extensions are parsed but not evaluated.

How to use

  1. Add user-agent groups and allow/disallow rules.
  2. Or paste an existing file to check a path.
  3. Copy the generated file or read the verdict.

What it supports

Matches Google-style longest-match rules with * wildcards; multiple groups are unioned. A pass here means 'not disallowed' — it cannot test whether crawlers actually obey it.

Related tools

Related guides