# robots.txt for innovacion.ai # https://innovacion.ai # # Policy: open by default. We WANT to be discoverable by traditional search # engines AND by LLM-based answer engines (ChatGPT, Claude, Gemini, Perplexity, # etc.). Increasing share of B2B research happens inside LLMs — being absent # from their indices is worse for us than being scraped. # # Last updated: 2026-05-09 (Sprint 9 · Phase AEO/GEO) User-agent: * Allow: / # Legal pages are listed in the sitemap (for Search Console visibility) # but each page emits at . # No need to Disallow here. # --------------------------------------------------------------------------- # AI / LLM crawlers — EXPLICITLY ALLOWED # --------------------------------------------------------------------------- # Each is allowed with a separate User-agent block so future per-bot tuning # (e.g. crawl-delay, path exclusion) is one-line. Order alphabetical. # Anthropic / Claude — training + ClaudeBot indexing (Claude with web search) User-agent: anthropic-ai Allow: / User-agent: ClaudeBot Allow: / User-agent: claude-web Allow: / # Apple — Apple Intelligence training (separate from Applebot for search) User-agent: Applebot Allow: / User-agent: Applebot-Extended Allow: / # ByteDance / TikTok / Doubao User-agent: Bytespider Allow: / # Common Crawl — feeds many LLM training corpora (GPT, LLaMA, Mistral, etc.) User-agent: CCBot Allow: / # Cohere User-agent: cohere-ai Allow: / User-agent: cohere-training-data-crawler Allow: / # Diffbot — used by enterprise LLMs for structured extraction User-agent: Diffbot Allow: / # Meta / Facebook AI training User-agent: FacebookBot Allow: / User-agent: Meta-ExternalAgent Allow: / # Google — Gemini + AI Overviews training (separate from Googlebot for search). # Google-Extended controls inclusion in Gemini training and AI Overviews; without # an explicit Allow, the default is allow but we make it explicit so it survives # any policy change. User-agent: Google-Extended Allow: / # Mistral User-agent: MistralAI-User Allow: / # OpenAI — three distinct user agents: # GPTBot : training crawler # OAI-SearchBot : ChatGPT search results indexer (citations in responses) # ChatGPT-User : on-demand fetch when a user clicks a link in ChatGPT User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / # Perplexity — two agents: training + on-demand user fetch User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / # You.com / YouBot User-agent: YouBot Allow: / # --------------------------------------------------------------------------- # Sitemaps — primary discovery for both classic search and LLM crawlers # --------------------------------------------------------------------------- Sitemap: https://innovacion.ai/sitemap-index.xml # llms.txt index — emerging standard (llmstxt.org) for LLM-friendly content # discovery. Pointer here lets crawlers find it deterministically. # https://innovacion.ai/llms.txt