# AI crawler rules for robots.txt. From the free course AEO for B2B, zrev.ai/learn/aeo, lesson 2.
# Decide each group on purpose, then merge these lines into https://www.yoursite.com/robots.txt.
# robots.txt is a request that well-behaved crawlers follow. Your CDN or firewall must ALSO let
# them in: test with  curl -s -o /dev/null -w "%{http_code}" -A "GPTBot" https://www.yoursite.com/

# Search: build the indexes assistants search when they answer. Most B2B companies allow these.
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# User-triggered: open a page because a person asked an assistant just now. Usually allowed.
User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Perplexity-User
Allow: /

# Training: collect pages for future models. Allowing them helps future models know you;
# blocking them does not remove you from search-based answers. Your call.
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Allow: /

# Keep private areas private for everyone.
User-agent: *
Disallow: /admin/
Disallow: /app/

Sitemap: https://www.yoursite.com/sitemap.xml