Tools
AI Crawler Access Checker
Paste a URL to check OAI-SearchBot, PerplexityBot, Googlebot, Bingbot, and Google-Extended against the site's robots.txt rules, and to see whether a sitemap is published.
Worked example: checking this site
Enter https://www.visiscan.app. The checker fetches /robots.txt, evaluates each named crawler against the public URL, and reports the matching allow or disallow rule. A useful result shows search crawlers allowed while private paths such as /api and /dashboard stay blocked.
What does this crawler check mean?
An allowed result means robots.txt does not block that user agent from the tested public URL. It does not guarantee indexing, citation, or recommendation. Search systems still decide whether a page is useful, trustworthy, and retrievable for a query.
Search bots and training bots are different
OAI-SearchBot supports ChatGPT search discovery; PerplexityBot supports Perplexity search; Googlebot and Bingbot feed their search indexes. Google-Extended is a separate control for some Google generative AI uses. GPTBot, ClaudeBot, and CCBot are training-oriented crawlers, so a publisher can block them without treating that choice as a failed search-access check.
Keep private routes blocked for every bot. Then verify the production file, sitemap, canonical URL, and indexing status. See the official OpenAI crawler documentation and Perplexity crawler documentation for current user-agent details.
Before you run a full visibility scan
If AI crawlers cannot reach your public pages, VisiScan's signal auditor will flag this — and it will pull your Readiness score down. Fix crawler access first, then run the scan. The checker above tells you whether each bot can reach your URL right now, without making any AI model calls. For the full checklist on crawler access, indexing, and getting cited by answer engines, see how to get your business cited by ChatGPT.
Questions about AI crawler access
- What does the AI crawler checker test?
- It reads your robots.txt the way a crawler does and reports whether the named discovery agents — GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, Googlebot, and Bingbot — are allowed to fetch your public pages, and which paths are disallowed.
- How much does the AI crawler checker cost?
- Nothing. It is free, needs no signup, and runs as many checks as you need against any public site.
- Which engines does it cover?
- The OpenAI crawlers behind ChatGPT, Anthropic's crawlers behind Claude, PerplexityBot, Google-Extended for Gemini and AI Overviews, and the ordinary Googlebot and Bingbot search crawlers.
- Does allowing these crawlers mean AI will cite my business?
- No. Crawler access is a precondition, not a cause. If a crawler is blocked your pages cannot be used at all, but being reachable still leaves relevance, authority, content quality, and the engine's own retrieval process to decide whether you are named.
- What are the checker's limits?
- It reads robots.txt and the server-rendered HTML only. It cannot see blocks applied at the firewall, CDN, or bot-management layer, which can refuse a crawler even when robots.txt allows it. If robots.txt looks correct but you are still not being crawled, check those layers next.
- Should I block AI crawlers?
- That is a business decision, not a technical one. Blocking training crawlers protects content from model training but also removes you from the answer surfaces those companies operate. Content Signals in robots.txt let you separate the two: allow crawling for search and answers while declining training use.