Understand What Blocking an AI Training Crawler Does

AI Readiness Analyzer

What is this warning?

Your robots.txt blocks a crawler that collects pages to train a model. This is a legitimate editorial choice and it costs you no traffic, so it is reported for your awareness rather than as a defect. It is worth confirming the block is deliberate, and that you have not accidentally blocked the retrieval crawlers alongside it.

How to Fix This Issue

How to Fix It

The Problem

A rule disallows a training crawler such as GPTBot, CCBot, Google-Extended or Applebot-Extended:

User-agent: GPTBot
Disallow: /

The Solution

If the block is intended, leave it. Check only that your retrieval crawlers are still allowed, since those are the ones that bring traffic:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Why This Works

Training and retrieval are separate user agents. Keeping them separate lets you decline to be training data while still being quotable and linkable in assistant answers.

SEO Impact

This issue can affect your site's search engine rankings and user experience. Addressing it promptly helps ensure optimal performance and visibility in search results.

Automatic Detection

bseoa automatically checks for this warning during site analysis, along with hundreds of other technical SEO issues.

Other AI Readiness Analyzer issues

Ready to Unlock Your Site's Full SEO Potential?

Choose the license that fits your needs and start getting the deep, actionable insights you deserve.