AI Crawler Permissions Checker

Verify if GPTBot, ClaudeBot, PerplexityBot, and Apple Intelligence can access your website. Test exact URL path permissions to prevent accidental de-indexing in AI search answers.

GEO Ingestion Readiness
100%

GPTBot, ClaudeBot & PerplexityBot accessibility

Training Scrapers Shielded
3/3

CCBot, Bytespider & Google-Extended blocked

Test Target URL Path
Your robots.txt File

Crawler Verdict on "/docs/api" (9 Bots Audited)

RFC 9309 Parser
Googlebot(Google Search)Traditional Search

Primary web indexing and search SERP listings

Allowed on '/docs/api' (Restricted on 2 other paths).

ALLOWED
Bingbot(Microsoft Bing)Traditional Search

Bing Search and Copilot grounding data

Allowed on '/docs/api' (Restricted on 1 other paths).

ALLOWED
GPTBot(OpenAI / SearchGPT)GEO Retrieval

ChatGPT real-time web retrieval & answers

Allowed on '/docs/api' (Restricted on 2 other paths).

ALLOWED
PerplexityBot(Perplexity AI)GEO Retrieval

Conversational search citations & synthesis

Allowed on '/docs/api' (Restricted on 1 other paths).

ALLOWED
ClaudeBot(Anthropic Claude)GEO Retrieval

Claude web browsing and artifact reference retrieval

Allowed on '/docs/api' (Restricted on 1 other paths).

ALLOWED
Applebot-Extended(Apple Intelligence)GEO Retrieval

Siri conversational intelligence and model training

Unrestricted crawl access permitted.

ALLOWED
Google-Extended(Google Gemini)Training Scraper

Gemini standalone model training corpus

Explicitly blocked across entire domain by 'Disallow: /'.

BLOCKED
CCBot(Common Crawl)Training Scraper

Open foundation dataset collection for AI researchers

Explicitly blocked across entire domain by 'Disallow: /'.

BLOCKED
Bytespider(ByteDance / TikTok)Training Scraper

ByteDance AI training and content scraping

Explicitly blocked across entire domain by 'Disallow: /'.

BLOCKED

The New Web Landscape: Search Indexing vs. AI Training Ingestion

In the early days of the web, webmasters configured robots.txt files exclusively for search engine crawlers like Googlebot and Bingbot. Today, web traffic is increasingly driven by conversational answers from Large Language Models.

However, many webmasters hastily added User-agent: * Disallow: / to legacy staging subdirectories or implemented blanket AI blocking directives without realizing they were blinding ChatGPT Search and Perplexity from citing their brand in real-time answers. Auditing path-level permissions ensures you maintain full control over training ingestion while securing conversational search citations.

Comprehensive 2026 AI Agent & Crawler Taxonomy

Bot User-AgentParent EntityClassificationDirect Traffic PotentialRecommended Policy
GPTBotOpenAIGEO RetrievalVery High (ChatGPT Answers)Allow
PerplexityBotPerplexity AIGEO RetrievalVery High (Footnote Links)Allow
ClaudeBotAnthropicGEO RetrievalHigh (Claude Citations)Allow
Applebot-ExtendedAppleGEO RetrievalHigh (Apple Intelligence)Allow
Google-ExtendedGoogleTraining ScraperZero (Training Only)Optional Disallow
CCBotCommon CrawlTraining ScraperZero (Archival)Disallow
BytespiderByteDanceTraining ScraperZero (Bandwidth Heavy)Disallow

Specific Record Precedence

Under RFC 9309, declaring User-agent: GPTBot Allow: / overrides any restrictive User-agent: * Disallow: / rules for OpenAI. If you want broad crawler protection while maintaining AI visibility, declare specific allowed blocks for retrieval agents above the wildcard block.

Preserving Server Egress Bandwidth

Bulk training scrapers crawl without regard for server latency, often consuming gigabytes of transfer bandwidth per day on unindexed endpoints. Explicitly blocking CCBot and Bytespider protects backend server resources without sacrificing organic SEO traffic.

AI Crawl Governance Checklist

✓Allow GPTBot, ClaudeBot, and PerplexityBot on public marketing and documentation pages
✓Disallow /admin/, /private/, /checkout/, and API keys across all User-agents
✓Block heavy non-referral training scrapers (CCBot, Bytespider) to preserve server load
✓Ensure your robots.txt returns an HTTP 200 OK status without intermediate redirects
✓Host an authoritative /llms.txt file to guide AI models to clean Markdown endpoints
✓Re-test robots.txt policies using our live path simulator before each production deploy

Frequently Asked Questions

Technical answers regarding AI search crawling, training data scrapers, and robots.txt directives.

Why do AI search engines require specific permissions in robots.txt?

Generative search engines like ChatGPT Search and Perplexity AI strictly respect RFC 9309 robots.txt directives before visiting any webpage. If your domain or specific subdirectories (like `/docs` or `/blog`) are disallowed for `GPTBot` or `PerplexityBot`, the AI models are legally and technically prohibited from scraping your content, entirely preventing your website from being cited in generative answers.

What is the difference between GPTBot, ChatGPT-User, and CCBot?

`GPTBot` is OpenAI's automated crawler used for indexing web data and powering real-time search. `ChatGPT-User` is triggered on demand when an individual user asks ChatGPT to browse a specific link. `CCBot` is the crawler for Common Crawl, an independent non-profit data repository whose archival data dumps are used by numerous tech companies to train raw foundation models.

Can I allow AI search bots while blocking model training scrapers?

Yes. This is the gold standard of modern GEO strategy. By allowing `GPTBot`, `ClaudeBot`, and `PerplexityBot` while explicitly adding `Disallow: /` for `Google-Extended`, `CCBot`, and `Bytespider`, you guarantee conversational visibility and footnote citation clicks without allowing your proprietary copy to be swallowed into offline training datasets.

How does the RFC 9309 User-Agent precedence rule work?

Under RFC 9309, a specific User-agent record (e.g., `User-agent: GPTBot`) takes absolute precedence over the generic wildcard (`User-agent: *`). If a specific record exists for a bot, that bot completely ignores all Allow and Disallow rules declared under the wildcard block.

What happens if a website has no robots.txt file?

If a crawler receives an HTTP 404 Not Found response when requesting `/robots.txt`, RFC 9309 mandates that the crawler assume there are zero restrictions. All search bots and AI crawlers are permitted to crawl the entire domain.

Does blocking AI crawlers save server bandwidth?

Yes, significantly. AI training scrapers (such as ByteDance's Bytespider or Common Crawl) frequently crawl aggressively, making thousands of rapid HTTP requests per minute. Blocking bulk training crawlers preserves server CPU, lowers database load, and cuts cloud egress bandwidth costs.

How can I verify if an AI crawler is spoofing its user-agent header?

Legitimate search bots and AI agents provide verifiable IP addresses. You can perform Reverse DNS lookups (rDNS) on incoming requests: a legitimate request from Googlebot will resolve back to a `.googlebot.com` domain, and OpenAI requests resolve back to official OpenAI CIDR blocks.

Where should our robots.txt file be hosted?

It must be hosted at the root of your domain and protocol: `https://yourdomain.com/robots.txt`. Subdirectory locations like `https://yourdomain.com/assets/robots.txt` are completely ignored by RFC 9309 compliant web crawlers.

Need Automated Continuous Audits?

Upgrade from Single-Check to Automated DAST Vulnerability Scanner

Full-scale active fuzzing for OWASP Top 10, SQLi, XSS, SSRF & BOLA. Scale with automated schedules, Slack alerts, 150+ monthly credits, and compliance PDF exports on our paid tiers.

Headless Chromium SPA crawlerActive OWASP Top 10 fuzzingAutomated SQLi & XSS verification
Explore Paid Plans (from $9/mo)Launch Free Instant DAST Scan

Expand Your Search & Security Arsenal

WeSecureX provides free client tools to audit search engine indexing, harden server headers, and optimize generative AI discovery.

Robots.txt & AI Crawler RulesLLMs.txt GeneratorRAG Chunkability ScorerView All Tools