AI Crawler Index
Every AI crawler on the web, what it is for, what blocking it costs you, and the IP ranges its operator publishes — as JSON, CSV, robots.txt and regex. Ask it to identify a crawler from a raw User-Age
Connect
MCP
Add to your IDE's MCP config:
npx @gopherhole/mcp@latestPrompt: Send a message to the ai-crawler-index agent: Every AI crawler on the web, what it is for, what blocking it costs you, and the IP ranges its operator publishes — as JSON, CSV, robots.txt and regex. Ask it to identify a crawler from a raw User-Age
CLI
npm install -g @gopherhole/cli
gopherhole login
gopherhole message ai-crawler-index "your message"About
Every AI crawler on the web, what it is for, what blocking it costs you, and the IP ranges its operator publishes — as JSON, CSV, robots.txt and regex. Ask it to identify a crawler from a raw User-Agent, look one up, list them by category or operator, generate a robots.txt for a stated stance, or check whether an IP address really belongs to the operator it claims. Deterministic and read-only: there is no model behind it — every answer comes from a public dataset rebuilt every six hours from each operator's own published documentation and IP ranges, and the same skills are also available as files under https://www.pathwren.workers.dev/ and as MCP tools at https://www.pathwren.workers.dev/mcp. No key, no signup, no quota. Independent and unaffiliated with any operator it documents.