Your robots.txt blocks a crawler that builds the index an AI assistant cites from. This is not the same as blocking a training crawler: retrieval crawlers fetch pages so the assistant can quote and link them, in real time, in answers to real questions. Blocking one removes your site from those answers entirely, and it is usually done by accident, by copying a blanket rule meant for training bots.
A robots.txt rule disallows a retrieval crawler such as OAI-SearchBot, Claude-SearchBot or PerplexityBot:
User-agent: OAI-SearchBot
Disallow: /Allow the retrieval crawlers while keeping whatever policy you want for training crawlers. They are separate user agents precisely so you can treat them differently:
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# Still opt out of model training if you want to:
User-agent: GPTBot
Disallow: /Retrieval crawlers are how an assistant finds a page worth citing. Allowing them puts your content in front of people at the moment they are asking about your subject, with a link back to you, which is the same bargain as appearing in a search result.
This issue can affect your site's search engine rankings and user experience. Addressing it promptly helps ensure optimal performance and visibility in search results.
bseoa automatically checks for this warning during site analysis, along with hundreds of other technical SEO issues.
Choose the license that fits your needs and start getting the deep, actionable insights you deserve.