Robots.txt & AI Search Crawler Generator

Build RFC 9309 compliant crawling rules. Balance traditional search engine indexation with Generative AI (GEO) discovery for OpenAI GPTBot, ClaudeBot, PerplexityBot, and Apple Intelligence.

Quick Policy Presets:

Disallowed Paths (4)

RFC 9309 Path Prefix Match
/api//admin//private//checkout/

AI Bots & Generative Retrieval Agents (2026 Standards)

AEO & GEO

Toggle specific AI retrieval agents. Allowing GPTBot, ClaudeBot, and PerplexityBot permits generative AI engines to summarize and cite your pages in conversational answers.

GPTBot(GPTBot)

OpenAI ChatGPT and SearchGPT retrieval agent

PerplexityBot(PerplexityBot)

Perplexity AI conversational search engine crawler

ClaudeBot(ClaudeBot)

Anthropic Claude web search and document agent

Applebot-Extended(Applebot-Extended)

Apple Intelligence generative training & Siri search

Google-Extended(Google-Extended)

Google Gemini standalone model training data ingestion

CCBot(CCBot)

Common Crawl foundation corpus collector

Bytespider(Bytespider)

ByteDance AI training and content scraping agent

XML Sitemaps & Crawl Rate

https://wesecurex.com/sitemap.xml
Live Rule Simulator & Path TesterInstant Verdict
Googlebot
BLOCKED(Disallow: /admin/)
Bingbot
BLOCKED(Disallow: /admin/)
GPTBot
BLOCKED(Disallow: /admin/)
PerplexityBot
BLOCKED(Disallow: /admin/)
ClaudeBot
BLOCKED(Disallow: /admin/)
Applebot-Extended
BLOCKED(Disallow: /admin/)
Google-Extended
BLOCKED(Disallow: /)
CCBot
BLOCKED(Disallow: /)
Bytespider
BLOCKED(Disallow: /)
Generated robots.txt Preview

The New Paradigm: Search Indexing vs. AI Training Ingestion

Until recently, webmasters managed robots.txt almost exclusively for search engine web crawlers like Googlebot and Bingbot. With the rapid ascendancy of Large Language Models (LLMs) and conversational search, webmasters must now make granular decisions regarding two distinct categories of automated agents:

  • Real-Time AI Retrieval Agents (GEO & AEO): Bots such as GPTBot, PerplexityBot, and ClaudeBot crawl websites dynamically to cite real-time data in response to user queries. Blocking these agents removes your brand from conversational search answers.
  • Bulk Foundation Model Training Scrapers: Bots such as CCBot (Common Crawl), Bytespider, and Google-Extended archive massive troves of web text to train future offline foundation model weights without providing immediate referral click-through traffic.

2026 Web Crawler & AI Agent Governance Taxonomy

User-Agent TokenOperating EntityPrimary PurposeReferral Traffic PotentialRecommended Policy
GooglebotGooglePrimary web indexing & DiscoverExtremely HighAllow / with path shields
BingbotMicrosoftBing Search & Copilot groundingHighAllow / with path shields
GPTBotOpenAIChatGPT Search & live answer citationVery High (Conversational)Allow for GEO visibility
PerplexityBotPerplexity AILive academic & factual search indexingHigh (Direct Footnote Links)Allow for GEO visibility
ClaudeBotAnthropicClaude web browsing & artifact retrievalModerate to HighAllow for GEO visibility
Google-ExtendedGoogleGemini model training data collectionZero (Training Only)Optional Disallow
CCBotCommon CrawlPublic archive data dump scrapingZero (Archival)Disallow (Saves bandwidth)

RFC 9309 Wildcard & Anchor Rules

Under RFC 9309, paths use simple prefix matching by default. Disallow: /admin matches /admin, /admin/, and /administrator.php. To target exact file extensions across all directories, use wildcards and the end-of-URL anchor: e.g. Disallow: /*.pdf$ blocks all PDF documents while allowing HTML pages.

Crawl Budget & Server Load Preservation

Aggressive bot crawling can trigger CPU spikes and database lockouts if bots index unindexed pagination loops, session IDs, or internal search query parameters. Disallowing patterns like Disallow: /*?sort=* and Disallow: /search?* prevents crawler traps and shields backend server resources.

Production Robots.txt Deployment Checklist

✓Serve file strictly at https://yourdomain.com/robots.txt with Content-Type: text/plain
✓Ensure HTTP response returns 200 OK (never 301/302 redirects or 404/500 errors)
✓Keep total file size under 500 KB as required by RFC 9309 specifications
✓Declare absolute HTTPS URLs for all XML Sitemaps
✓Never place confidential URL paths (e.g., hidden admin paths) in robots.txt
✓Verify rules with Google Search Console Robots.txt Tester and live path simulator

Frequently Asked Questions

Technical answers on robots exclusion, AI agent permissions, and crawl mechanics.

What is RFC 9309 and why is it crucial for modern robots.txt files?

RFC 9309 is the formal IETF Internet Standard for the Robots Exclusion Protocol, ratified in 2022. It standardizes crawler behavior across all major search engines and AI agents, formalizing path prefix matching, wildcard handling (*), end-of-URL markers ($), and establishing that robots.txt files must be served with an HTTP 200 status code at the root path (/robots.txt) within a 500 KB limit.

Should I block or allow AI search crawlers like GPTBot and PerplexityBot?

If your objective is organic discoverability in AI answers (Generative Engine Optimization or GEO), allowing GPTBot, ClaudeBot, and PerplexityBot is essential. These conversational engines cite and summarize pages in real-time when answering user queries. However, if you wish to prevent your proprietary text or documentation from being used in offline foundation model training datasets, you can block CCBot, Bytespider, and Google-Extended while keeping retrieval agents open.

Does Disallow in robots.txt prevent a page from appearing in Google search results?

No. This is one of the most common SEO misconceptions. A 'Disallow' directive prevents search engine crawlers from downloading and rendering the page content. However, if external websites link to that URL with descriptive anchor text, Google can still index the URL and display it in SERPs without a meta description snippet. To completely prevent indexation, use a 'noindex' robots meta tag or 'X-Robots-Tag: noindex' HTTP response header.

How does robots.txt affect crawl budget on enterprise websites?

On websites with tens of thousands of URLs, search bots have finite daily crawl capacities (Crawl Budget). By disallowing faceted navigation filters, duplicate sorting parameters, internal search result pages, and heavy legacy staging endpoints, you funnel crawler bandwidth directly into your revenue-driving, canonical product and article pages.

What happens if a robots.txt file returns an HTTP 500 server error?

If a crawler receives an HTTP 5xx Internal Server Error when requesting /robots.txt, it treats the entire website as temporarily inaccessible and halts crawling to prevent overloading the server. If the 5xx error persists for multiple days, search engines may begin dropping pages from the index.

Can I declare multiple XML sitemaps in a single robots.txt file?

Yes. You can declare multiple Sitemap directives on separate lines (e.g., Sitemap: https://example.com/sitemap-articles.xml, Sitemap: https://example.com/sitemap-products.xml). Search engines parse all declared sitemap locations independently.

Do search engine bots respect the Crawl-delay directive?

Bingbot, Yandex, and Baidu respect the Crawl-delay directive to throttle request cadence. However, Googlebot ignores Crawl-delay in robots.txt; Google adjusts crawl frequency dynamically based on server response latency, though you can configure maximum crawl rate in Google Search Console.

Where must the robots.txt file be hosted?

Robots.txt must be hosted strictly at the root of the domain and protocol: e.g., https://example.com/robots.txt. Files placed in subdirectories (like https://example.com/assets/robots.txt) are completely ignored by RFC 9309 compliant crawlers.

Need Automated Continuous Audits?

Upgrade from Single-Check to Automated DAST Vulnerability Scanner

Full-scale active fuzzing for OWASP Top 10, SQLi, XSS, SSRF & BOLA. Scale with automated schedules, Slack alerts, 150+ monthly credits, and compliance PDF exports on our paid tiers.

Headless Chromium SPA crawlerActive OWASP Top 10 fuzzingAutomated SQLi & XSS verification
Explore Paid Plans (from $9/mo)Launch Free Instant DAST Scan

Complete Your Technical SEO & Security Stack

WeSecureX provides zero-login client tools to audit search engine indexing, harden server headers, and optimize generative AI discovery.

AI Crawler Permissions CheckerLLMs.txt Generator.htaccess & Nginx ConfigsView All Tools