---
title: "Noindex: Meta Robots vs X-Robots-Tag vs robots.txt"
description: "Meta robots, X-Robots-Tag and robots.txt compared: which ones noindex a page, why robots.txt can't, what happens when they conflict, and how to check."
image: "https://www.blackseoanalyzer.com/static/images/black-seo-analyzer-og-image.png"
canonical: "https://www.blackseoanalyzer.com/en/blog/noindex-directives-compared"
language: "en"
---

# Noindex Directives Compared: Meta Robots vs X-Robots-Tag vs robots.txt

By [Seth Black](https://www.blackseoanalyzer.com/en/about) · Published November 01, 2026

There are two noindex directives that work: the robots meta tag in a page’s HTML and the `X-Robots-Tag` HTTP header. Both tell search engines not to show a URL in results, and Google treats them the same. robots.txt is the one people confuse with them: it controls crawling, not indexing, and a URL blocked in robots.txt can still be indexed because Google never sees its noindex.

## What are the noindex directives?

| | Robots meta tag | X-Robots-Tag header | robots.txt `Disallow` |
| --- | --- | --- | --- |
| Where it lives | `<head>` of the HTML | HTTP response header | `/robots.txt` |
| Controls | Indexing and snippets | Indexing and snippets | Crawling only |
| Works on PDFs, images | No | Yes | Yes, for crawling |
| Removes a URL from results | Yes, after recrawl | Yes, after recrawl | No |
| Page must be crawlable | Yes | Yes | Not applicable |
| Site-wide in one line | No, every template | Yes, server config | Yes |

The meta tag:

```html
<meta name="robots" content="noindex">
```

The header:

```http
X-Robots-Tag: noindex
```

robots.txt:

```text
User-agent: *
Disallow: /internal-search/
```

## Why doesn’t robots.txt noindex a page?

Because a disallowed URL is never fetched, so whatever the page says is never read. Google’s documentation on [blocking indexing](https://developers.google.com/search/docs/crawling-indexing/block-indexing) is explicit: if robots.txt blocks the page, “the crawler will never see the `noindex` rule, and the page can still appear in search results.” Its [robots.txt introduction](https://developers.google.com/search/docs/crawling-indexing/robots/intro) adds that if other pages link to a blocked URL with descriptive text, Google could index the URL without visiting it, and the result shows no description.

There’s also no `noindex` rule in robots.txt. Google’s [robots.txt specification](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec) lists the fields it supports as `user-agent`, `allow`, `disallow` and `sitemap`, and says other rules are ignored. A `Noindex:` line in robots.txt does nothing in Google.

So the rule is:

- **To keep a URL out of search results:** allow crawling, and send `noindex` in the meta tag or the header.
- **To stop crawlers from spending requests on a section:** `Disallow` it in robots.txt, and accept that its URLs may still appear, bare, if linked.
- **To remove something that’s already indexed:** `noindex` first, wait until it drops out, and only then add a `Disallow` if you still want one.

## Meta robots or X-Robots-Tag: which should you use?

For HTML pages, use whichever your team can keep correct. They’re equivalent. The meta tag is easier to see when you view source; the header is easier to apply to a whole directory or host.

Use the header when:

- The file isn’t HTML: PDFs, Word documents, images, video, feeds.
- You need one rule for a whole staging or preview host.
- You control the web server but not the application’s templates.

I wrote a full post on the header’s syntax and server examples: [X-Robots-Tag: how to use the HTTP header](https://www.blackseoanalyzer.com/en/blog/x-robots-tag). The short nginx version for a staging host:

```nginx
server {
    server_name staging.example.com;
    add_header X-Robots-Tag "noindex, nofollow" always;
}
```

## What happens when directives conflict?

Google’s [robots meta tag documentation](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag) says: “In the case of conflicting robots rules, the more restrictive rule applies.” A page with `<meta name="robots" content="index">` and `X-Robots-Tag: noindex` is noindexed. So is a page whose meta tag says `index, noindex`, which happens more often than you’d think when a plugin and a theme both write the tag.

You can target Google alone with `<meta name="googlebot" content="noindex">`. Other search engines ignore that one and keep reading `name="robots"`.

## Does noindex work if JavaScript adds or removes it?

Don’t rely on it. Google’s [JavaScript SEO basics](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics) says that when Google encounters `noindex` it “may skip rendering and JavaScript execution”, and that using JavaScript to change or remove the tag “may not work as expected”. If a page should be indexed, the server-rendered HTML must not contain `noindex`, even if your app removes it a moment later.

## How do you check which directive a URL is sending?

```bash
# the header
curl -s -D - -o /dev/null https://example.com/page | grep -i x-robots-tag

# the meta tag in the raw HTML
curl -s https://example.com/page | grep -io '<meta[^>]*name="\(robots\|googlebot\)"[^>]*>'
```

Then check robots.txt for the path. Our free [robots.txt tester](https://www.blackseoanalyzer.com/en/tools/robots-txt-tester) fetches a site’s robots.txt and tells you whether a given URL is allowed for Googlebot and other crawlers, which is the step people skip before wondering why their noindex isn’t working.

For Google’s own view, use URL Inspection in Search Console. “Excluded by ‘noindex’ tag” covers both the meta tag and the header. “Indexed, though blocked by robots.txt” is the robots.txt trap described above.

## What bseoa reports

bseoa reads the robots meta tag on every page it crawls. It reports pages that carry `noindex`, so a stray one on a template shows up across a crawl instead of in next month’s traffic report, and it flags tags that contradict themselves, like [conflicting index and noindex](https://www.blackseoanalyzer.com/en/technical-seo/metadata-analyzer/conflicting-index-and-noindex). It doesn’t read the `X-Robots-Tag` header, so check headers with `curl` as above. By default the crawler also obeys robots.txt, which means it won’t fetch disallowed URLs either; that matches what Googlebot can see.

## Common noindex mistakes

**Disallow plus noindex on the same URL.** The noindex is never seen. Pick noindex.

**Noindex on the only pages that link to something important.** Paginated archives and filtered listings are often the only path to older products or posts. If you noindex them, make sure the pages behind them are also linked from indexable pages, so discovery doesn’t depend on URLs you’ve asked Google to drop.

**Leaving staging’s noindex in production.** Check a few production URLs’ headers and HTML after every deploy.

**Noindexing instead of fixing duplicates.** For duplicate URLs, a canonical tag usually works better than noindex. See [what is a non-canonical URL](https://www.blackseoanalyzer.com/en/blog/what-is-a-non-canonical-url).

-Sethers

About the author

[Seth Black](https://www.blackseoanalyzer.com/en/about)

Seth is a software engineer and engineering leader who builds bseoa, a Rust-based technical SEO crawler with a GUI and CLI. He writes about the crawling, rendering and indexing problems he runs into on real sites.

[LinkedIn](https://www.linkedin.com/in/seth-black-tx/) · [GitHub](https://github.com/sethblack) · [YouTube](https://www.youtube.com/@SethBlack)

[Back to Blog](https://www.blackseoanalyzer.com/en/blog)

## Discover hundreds of SEO Issues in Seconds

Without Monthly Subscriptions

Comprehensive technical SEO analysis powered by ML and 16 specialized modules. Optional AI-powered insights from Claude, GPT-4, or Gemini. Get actionable insights in seconds, and never pay monthly fees again.

[Download Free Trial](https://www.blackseoanalyzer.com/en/free-trial)

I use AI to generate images for my posts and for general editing, updates, and ironically SEO purposes. I used to draw all of the images for my personal blog (taleas) myself, but as the volume of content I produce has increased, I've turned to AI tools to help create visuals that complement my writing. I go out of my way to generate images that look strange, and don't represent real people. If you ever want to chat about my use of AI, please reach out.

```json
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://www.blackseoanalyzer.com/#organization","name":"Fiscus Technology, LLC","url":"https://www.blackseoanalyzer.com/","logo":{"@type":"ImageObject","url":"https://www.blackseoanalyzer.com/static/images/black-seo-analyzer.png"},"sameAs":["https://sethserver.com/","https://github.com/sethblack"],"founder":{"@id":"https://www.blackseoanalyzer.com/#seth"}},{"@type":"WebSite","@id":"https://www.blackseoanalyzer.com/#website","name":"bseoa","url":"https://www.blackseoanalyzer.com/","publisher":{"@id":"https://www.blackseoanalyzer.com/#organization"}},{"@type":"Person","@id":"https://www.blackseoanalyzer.com/#seth","name":"Seth Black","url":"https://www.blackseoanalyzer.com/about","jobTitle":"Creator of bseoa","sameAs":["https://www.linkedin.com/in/seth-black-tx/","https://github.com/sethblack","https://www.youtube.com/@SethBlack","https://sethserver.com/"]},{"@type":"BlogPosting","@id":"https://www.blackseoanalyzer.com/en/blog/noindex-directives-compared#article","headline":"Noindex Directives Compared: Meta Robots vs X-Robots-Tag vs robots.txt","name":"Noindex Directives Compared: Meta Robots vs X-Robots-Tag vs robots.txt","description":"Meta robots, X-Robots-Tag and robots.txt compared: which ones noindex a page, why robots.txt can't, what happens when they conflict, and how to check.","url":"https://www.blackseoanalyzer.com/en/blog/noindex-directives-compared","datePublished":"2026-11-01T05:21:24","dateModified":"2026-11-01T05:21:24","author":{"@id":"https://www.blackseoanalyzer.com/#seth"},"publisher":{"@id":"https://www.blackseoanalyzer.com/#organization"},"image":{"@type":"ImageObject","url":"https://www.blackseoanalyzer.com/static/images/black-seo-analyzer-og-image.png","width":1200,"height":630},"mainEntityOfPage":{"@type":"WebPage","@id":"https://www.blackseoanalyzer.com/en/blog/noindex-directives-compared"},"isPartOf":{"@id":"https://www.blackseoanalyzer.com/#website"},"inLanguage":"en-US","isAccessibleForFree":true,"wordCount":969},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://www.blackseoanalyzer.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://www.blackseoanalyzer.com/en/blog"},{"@type":"ListItem","position":3,"name":"Noindex Directives Compared: Meta Robots vs X-Robots-Tag vs robots.txt","item":"https://www.blackseoanalyzer.com/en/blog/noindex-directives-compared"}]}]}
```