Every AI crawler we check is disallowed. In almost every case this comes from a single blanket rule rather than a deliberate decision about each bot, and it means no assistant can read, quote or link your site. If you meant to opt out of model training only, this goes considerably further than that.
A blanket rule catches every user agent, including the retrieval crawlers that would otherwise send you traffic:
User-agent: *
Disallow: /Decide per purpose. Keep any paths you genuinely want private, allow the retrieval and user-action agents, and opt out of training separately if you wish:
User-agent: *
Disallow: /admin/
Disallow: /cart/
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /A blanket disallow treats a crawler that would cite you the same as one that would train on you. Separating them keeps the traffic while preserving whatever training policy you want.
This issue can affect your site's search engine rankings and user experience. Addressing it promptly helps ensure optimal performance and visibility in search results.
bseoa automatically checks for this warning during site analysis, along with hundreds of other technical SEO issues.
Choose the license that fits your needs and start getting the deep, actionable insights you deserve.