robots.txt Checker and AI Crawler Test
Validate robots.txt and see which search and AI crawlers are blocked.
Use it from code or an agent
curl "https://kitbag.cleartrust.cc/api/robots?domain=example.com&path=/private&user_agent=GPTBot"Also available as an MCP tool at https://kitbag.cleartrust.cc/mcp. Details