Google Search Console vs a Crawler: What Each Can and Cannot Tell You
By Seth Black · Published
Google Search Console tells you what Google has done with your site: which URLs it knows about, which it indexed and why it skipped the rest, and which queries brought clicks. A crawler tells you what your site actually contains right now: every reachable page, every link, every status code and tag, whether Google has seen it yet or not. You need both, because each one is blind exactly where the other one sees.
Google’s documentation quoted below was checked on 31 October 2026.
What can Search Console tell you that a crawler can’t?
Anything that happens inside Google.
- Indexing status. The Page indexing report “shows the Google indexing status of all URLs that Google knows about in your property,” with reasons such as “Crawled - currently not indexed,” “Discovered - currently not indexed,” “Duplicate, Google chose different canonical than user” and “Soft 404.” A crawler can see your canonical tag; only Google can tell you it ignored it.
- Search performance. Queries, clicks, impressions and position. No crawler has this.
- Google’s rendered view of a page. The URL Inspection live test shows a screenshot, the HTML, JavaScript console output and the resources Google loaded.
- Links Google found from other sites. A crawler of your own site can’t see links that live on someone else’s.
- Manual actions and security issues. Only Google knows if it has penalized you.
What can a crawler tell you that Search Console can’t?
Anything Google hasn’t found yet, and everything in full.
- Pages Google doesn’t know about. The Page indexing report only covers URLs Google has discovered. A new section with no internal links won’t appear at all. A crawl shows you what is reachable and what isn’t.
- Complete lists. Search Console’s examples are capped. The Page indexing help page says “the list of example URLs in the report is limited to 1,000 items” and “isn’t guaranteed to show all URLs in a given status.” A crawler lists every URL it reached.
- The current state. Google recrawls on its own schedule, and the help page says “Indexing is never instant.” A crawl you start now reflects the site as it is now, including the fix you deployed ten minutes ago.
- Staging sites. Search Console only covers live, verified properties. A crawler can audit staging before release.
- Every link with its anchor text and source page, so you can find which page links to a broken URL, not just that the URL is broken.
- Page-level checks Google doesn’t report: oversized images, render-blocking scripts, hreflang errors, mixed content, thin content and the rest.
How do Search Console and a crawler compare?
| Question | Search Console | Crawler |
|---|---|---|
| Is this page indexed? | Yes, authoritative | No |
| Why isn’t it indexed? | Yes, Google’s reason | Partly: finds noindex, robots.txt blocks, errors |
| Which queries bring clicks? | Yes | No |
| Every page on the site, today | No, only URLs Google knows, examples capped at 1,000 | Yes |
| Which page links to this broken URL? | Partly, via referring page in URL Inspection | Yes, every link and source |
| Staging or pre-release audit | No | Yes |
| What Google’s renderer saw | Yes, URL Inspection live test | Its own render, if it renders JavaScript |
| Backlinks from other sites | Yes, Links report | No |
| Speed of feedback after a fix | Days to weeks | Minutes |
| Cost | Free | Free tiers to enterprise contracts |
What are Search Console’s data limits?
These matter once you try to use it as a full inventory:
| Limit | Value | Source |
|---|---|---|
| Example URLs per Page indexing status | 1,000 | Page indexing help |
| 404 view | URLs with 404s “in the past month” | Same page |
| Search Analytics API rows per request | Default 1,000, maximum 25,000, paged with startRow; returns “top ones,” not all rows | API reference |
| URL Inspection API | 2,000 queries per day and 600 per minute, per site | API usage limits |
| URL Inspection in the web tool | Daily limits per property, separate for live tests | URL Inspection help |
The URL Inspection API is the closest thing to bulk indexing data: at 2,000 a day, checking 50,000 URLs takes 25 days. That’s why the practical workflow pairs the two tools.
How do you use Search Console and a crawler together?
- Crawl the site. Get the full list of reachable URLs with status codes, canonicals, noindex tags and inlinks.
- Export Search Console’s indexed and not-indexed examples, and the Performance report’s pages.
- Compare the lists.
| In crawl | In Search Console | What it usually means |
|---|---|---|
| Yes, indexable | Not indexed | Quality, duplication or crawl budget. Check Google’s reason. |
| Yes | Not present at all | Google hasn’t discovered it. Check internal links and the sitemap. |
| No | Indexed or getting clicks | Orphan page, or an old URL still in Google’s index. |
| Yes, noindex or blocked | Indexed | Google hasn’t recrawled since the change, or robots.txt blocks it from seeing the noindex. |
The third row is the one people miss. Pages that get impressions but that your crawler can’t reach are orphans; how to find orphan pages covers that comparison step by step.
- Fix, recrawl, then validate. Recrawl to confirm the fix is live, then use Search Console’s validation to ask Google to check. The help page notes Google “immediately checks a few pages” first, then works through the rest.
Which crawler pairs well with Search Console?
Any crawler that exports a full URL list works. Screaming Frog connects to the Search Console API directly. bseoa doesn’t integrate with Search Console; it crawls from your machine, renders JavaScript with headless Chrome when you ask it to, and exports every page to CSV or JSON for you to join against a Search Console export. It’s $279 once with no page limit, and the 14-day trial has every feature. If you work from Claude or ChatGPT, the connector can run the crawl and answer questions about it while you have Search Console open in another tab.
Further reading
Why your site audit tool is missing half your pages explains why a crawl and Search Console disagree on page counts. Crawler prices are on SEO crawler pricing.
-Sethers