Kitbag

robots.txt Checker and AI Crawler Test

Validate robots.txt and see which search and AI crawlers are blocked.

Use it from code or an agent
curl "https://kitbag.cleartrust.cc/api/robots?domain=example.com&path=/private&user_agent=GPTBot"

Also available as an MCP tool at https://kitbag.cleartrust.cc/mcp. Details