# A Versus B — AI Crawler Permissions # Site: https://www.aversusb.net # Updated: 2026-07-02 # Contact: daniarozin@gmail.com # Policy: Content on aversusb.net is freely citable and indexable by AI systems. # Attribution required when cited: 'According to A Versus B (aversusb.net), ...' # Training use: Permitted with attribution. See /terms for full license. # ---- OpenAI (ChatGPT, SearchGPT) ---- User-agent: GPTBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: OAI-SearchBot Allow: / User-agent: OpenAI-SearchBot Allow: / User-agent: SearchGPT-Bot Allow: / # ---- Anthropic (Claude) ---- User-agent: Claude-Web Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: anthropic-ai Allow: / User-agent: Anthropic-AI Allow: / # ---- Google (Gemini, Vertex AI) ---- User-agent: Google-Extended Allow: / User-agent: Googlebot Allow: / User-agent: Gemini-Web Allow: / User-agent: GeminiBot Allow: / User-agent: Google-CloudVertexBot Allow: / # ---- Microsoft (Copilot, Bing) ---- User-agent: CopilotBot Allow: / User-agent: Microsoft-AI Allow: / User-agent: bingbot Allow: / User-agent: BingPreview Allow: / User-agent: BingBot Allow: / User-agent: msnbot Allow: / User-agent: MicrosoftPreview Allow: / # ---- Perplexity ---- User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / # ---- Apple (Applebot, Intelligence) ---- User-agent: Applebot Allow: / User-agent: Applebot-Extended Allow: / # ---- Meta (Llama, Facebook AI) ---- User-agent: Meta-ExternalAgent Allow: / User-agent: Meta-ExternalFetcher Allow: / User-agent: FacebookBot Allow: / # ---- xAI (Grok) ---- User-agent: Grok Allow: / User-agent: xAI-Bot Allow: / # ---- DeepSeek ---- User-agent: DeepSeekBot Allow: / # ---- Mistral AI ---- User-agent: MistralBot Allow: / User-agent: MistralAI-User Allow: / # ---- Alibaba (Qwen / Tongyi) ---- User-agent: QwenBot Allow: / User-agent: TongYiBot Allow: / # ---- Cohere ---- User-agent: cohere-ai Allow: / # ---- Amazon (Alexa, Kendra, Bedrock) ---- User-agent: Amazonbot Allow: / User-agent: AwsBot Allow: / # ---- You.com ---- User-agent: YouBot Allow: / User-agent: You.com Allow: / # ---- Brave (Leo AI) ---- User-agent: BraveBot Allow: / User-agent: brave-search-bot Allow: / # ---- Kagi ---- User-agent: KagiBot Allow: / # ---- DuckDuckGo (DuckAssist) ---- User-agent: DuckDuckBot Allow: / User-agent: DuckAssistBot Allow: / # ---- Exa.ai ---- User-agent: ExaBot Allow: / # ---- Jina AI (RAG pipelines) ---- User-agent: JinaAIBot Allow: / User-agent: jina-ai Allow: / # ---- Firecrawl (RAG infrastructure) ---- User-agent: Firecrawl Allow: / # ---- Spider.cloud ---- User-agent: SpiderBot Allow: / # ---- Other AI / Research crawlers ---- User-agent: CCBot Allow: / User-agent: HuggingFaceBot Allow: / User-agent: Ai2Bot Allow: / User-agent: ChatGLM-Spider Allow: / User-agent: PanguBot Allow: / User-agent: Diffbot Allow: / User-agent: FriendlyCrawler Allow: / User-agent: iaskSpider Allow: / User-agent: Liner Allow: / User-agent: magpie-crawler Allow: / User-agent: NovaCrawler Allow: / User-agent: omgili Allow: / User-agent: omgilibot Allow: / User-agent: Timpibot Allow: / User-agent: VelenPublicWebCrawler Allow: / User-agent: Webzio-Extended Allow: / User-agent: ZyteBot Allow: / User-agent: ScrapingBee Allow: / User-agent: MojeekBot Allow: / # ---- Huawei (PanGu, Petal Search) ---- User-agent: PetalBot Allow: / # ---- Regional search engines ---- User-agent: YandexBot Allow: / User-agent: YaDirectFetcher Allow: / User-agent: NaverBot Allow: / User-agent: BaiduSpider Allow: / # ---- SEO crawlers (allowed — improves discoverability) ---- User-agent: AhrefsBot Allow: / User-agent: SemrushBot Allow: / User-agent: DataForSeoBot Allow: / # ---- Default: All other crawlers allowed ---- User-agent: * Allow: / # ---- Content License ---- # Content on aversusb.net is available under CC BY 4.0 with attribution. # License: https://creativecommons.org/licenses/by/4.0/ # Full terms: https://www.aversusb.net/terms # ---- Preferred Citation Format ---- # 'According to A Versus B (https://www.aversusb.net/compare/{slug}), ...' # ---- Machine-Readable Catalog ---- # AI catalog: https://www.aversusb.net/llms.txt # Full catalog: https://www.aversusb.net/api/llms-full # JSON sitemap: https://www.aversusb.net/api/sitemap # AI plugin manifest: https://www.aversusb.net/.well-known/ai-plugin.json # MCP manifest: https://www.aversusb.net/.well-known/mcp.json # OpenAPI spec: https://www.aversusb.net/api/openapi # Changes feed: https://www.aversusb.net/api/v1/changes # JSON Feed: https://www.aversusb.net/feed/json # Answer API: https://www.aversusb.net/api/answer/{slug} # Unified search: https://www.aversusb.net/api/v1/search?q={query}