Write and test a robots.txt

The job

robots.txt sits at the site root and tells crawlers what not to fetch. Two common jobs: keep a whole staging site out of every index, or hide one directory while the rest of the site stays crawlable.

Build it

In build mode, add a group with the agent name (*, or Googlebot), then Allow/Disallow rules. Disallow: / blocks everything; Disallow: /staging blocks one tree. The tool emits the file to paste at your root.

• Sitemap: https://example.com/sitemap.xml can be added as a site-wide line.
• Rules match by path prefix; the longest match wins, so Allow: /staging/public beats Disallow: /staging for that subtree.

Check a path

Switch to check mode and paste an existing robots.txt plus a path like /private/report.pdf. The verdict applies longest-match rules with * wildcards — the same resolution Googlebot uses — and shows which rule decided it.

Limits

robots.txt is advisory: well-behaved crawlers obey it, scrapers ignore it. Never use it to keep secrets — it is a public file. A 'allowed' verdict means 'not disallowed'; it cannot promise indexing.