Noindex Directives Compared: Meta Robots vs X-Robots-Tag vs robots.txt

There are two noindex directives that work: the robots meta tag in a page’s HTML and the X-Robots-Tag HTTP header. Both tell search engines not to show a URL in results, and Google treats them the same. robots.txt is the one people confuse with them: it controls crawling, not indexing, and a URL blocked in robots.txt can still be indexed because Google never sees its noindex.

What are the noindex directives?

	Robots meta tag	X-Robots-Tag header	robots.txt Disallow
Where it lives	<head> of the HTML	HTTP response header	/robots.txt
Controls	Indexing and snippets	Indexing and snippets	Crawling only
Works on PDFs, images	No	Yes	Yes, for crawling
Removes a URL from results	Yes, after recrawl	Yes, after recrawl	No
Page must be crawlable	Yes	Yes	Not applicable
Site-wide in one line	No, every template	Yes, server config	Yes

The meta tag:

<meta name="robots" content="noindex">

The header:

X-Robots-Tag: noindex

robots.txt:

User-agent: *
Disallow: /internal-search/

Why doesn’t robots.txt noindex a page?

Because a disallowed URL is never fetched, so whatever the page says is never read. Google’s documentation on blocking indexing is explicit: if robots.txt blocks the page, “the crawler will never see the noindex rule, and the page can still appear in search results.” Its robots.txt introduction adds that if other pages link to a blocked URL with descriptive text, Google could index the URL without visiting it, and the result shows no description.

There’s also no noindex rule in robots.txt. Google’s robots.txt specification lists the fields it supports as user-agent, allow, disallow and sitemap, and says other rules are ignored. A Noindex: line in robots.txt does nothing in Google.

So the rule is:

• To keep a URL out of search results: allow crawling, and send noindex in the meta tag or the header.
• To stop crawlers from spending requests on a section: Disallow it in robots.txt, and accept that its URLs may still appear, bare, if linked.
• To remove something that’s already indexed: noindex first, wait until it drops out, and only then add a Disallow if you still want one.

Meta robots or X-Robots-Tag: which should you use?

For HTML pages, use whichever your team can keep correct. They’re equivalent. The meta tag is easier to see when you view source; the header is easier to apply to a whole directory or host.

Use the header when:

• The file isn’t HTML: PDFs, Word documents, images, video, feeds.
• You need one rule for a whole staging or preview host.
• You control the web server but not the application’s templates.

I wrote a full post on the header’s syntax and server examples: X-Robots-Tag: how to use the HTTP header. The short nginx version for a staging host:

server {
    server_name staging.example.com;
    add_header X-Robots-Tag "noindex, nofollow" always;
}

What happens when directives conflict?

Google’s robots meta tag documentation says: “In the case of conflicting robots rules, the more restrictive rule applies.” A page with <meta name="robots" content="index"> and X-Robots-Tag: noindex is noindexed. So is a page whose meta tag says index, noindex, which happens more often than you’d think when a plugin and a theme both write the tag.

You can target Google alone with <meta name="googlebot" content="noindex">. Other search engines ignore that one and keep reading name="robots".

Does noindex work if JavaScript adds or removes it?

Don’t rely on it. Google’s JavaScript SEO basics says that when Google encounters noindex it “may skip rendering and JavaScript execution”, and that using JavaScript to change or remove the tag “may not work as expected”. If a page should be indexed, the server-rendered HTML must not contain noindex, even if your app removes it a moment later.

How do you check which directive a URL is sending?

# the header
curl -s -D - -o /dev/null https://example.com/page | grep -i x-robots-tag

# the meta tag in the raw HTML
curl -s https://example.com/page | grep -io '<meta[^>]*name="\(robots\|googlebot\)"[^>]*>'

Then check robots.txt for the path. Our free robots.txt tester fetches a site’s robots.txt and tells you whether a given URL is allowed for Googlebot and other crawlers, which is the step people skip before wondering why their noindex isn’t working.

For Google’s own view, use URL Inspection in Search Console. “Excluded by ‘noindex’ tag” covers both the meta tag and the header. “Indexed, though blocked by robots.txt” is the robots.txt trap described above.

What bseoa reports

bseoa reads the robots meta tag on every page it crawls. It reports pages that carry noindex, so a stray one on a template shows up across a crawl instead of in next month’s traffic report, and it flags tags that contradict themselves, like conflicting index and noindex. It doesn’t read the X-Robots-Tag header, so check headers with curl as above. By default the crawler also obeys robots.txt, which means it won’t fetch disallowed URLs either; that matches what Googlebot can see.

Common noindex mistakes

Disallow plus noindex on the same URL. The noindex is never seen. Pick noindex.

Noindex on the only pages that link to something important. Paginated archives and filtered listings are often the only path to older products or posts. If you noindex them, make sure the pages behind them are also linked from indexable pages, so discovery doesn’t depend on URLs you’ve asked Google to drop.

Leaving staging’s noindex in production. Check a few production URLs’ headers and HTML after every deploy.

Noindexing instead of fixing duplicates. For duplicate URLs, a canonical tag usually works better than noindex. See what is a non-canonical URL.

-Sethers