# Allow all search engine and AI crawlers User-agent: * Disallow: # --- AI answer engine / LLM crawlers --- # Intentionally allowed via the wildcard rule above (search, AI answers and # training). Listed here for audit purposes only; no Disallow rules apply. # GPTBot, OAI-SearchBot, ChatGPT-User (OpenAI) # ClaudeBot, Claude-SearchBot, Claude-User (Anthropic) # PerplexityBot, Perplexity-User (Perplexity) # Google-Extended (Google Gemini / AI Overviews) # Applebot, Meta-ExternalAgent, DuckAssistBot, MistralAI-User # CCBot (Common Crawl) # Bytespider (ByteDance) # Amazonbot (Amazon AI) # Access is managed in Cloudflare (AI Crawl Control: Search, Training and # Agents all set to Allow). # See also: /llms.txt for a curated, AI-agent-friendly summary of this site. Sitemap: https://btcompass.com/sitemap.xml Sitemap: https://btcompass.com/sitemap-dogs.xml