X-Robots-Tag: How to Use the HTTP Header

The X-Robots-Tag is the robots meta tag’s less famous sibling. It does the same job, but in an HTTP response header instead of the HTML. That makes it the only way to control indexing for PDFs, images and other files that don’t have a <head>, and the easiest way to noindex an entire staging site from the server config.

What X-Robots-Tag is

It’s an HTTP response header that tells search engines how to index and display a URL:

HTTP/1.1 200 OK
Content-Type: application/pdf
X-Robots-Tag: noindex, nofollow

Any directive that works in a robots meta tag works in the header. These two are equivalent for an HTML page:

<meta name="robots" content="noindex, nofollow">

X-Robots-Tag: noindex, nofollow

When to use it instead of the meta tag

• Non-HTML files. PDFs, Word documents, images, videos and feeds can’t carry a meta tag. The header is the only way to noindex them.
• Whole sites or directories. One line in the server config covers every URL, so you don’t depend on every template getting it right. Staging and preview environments are the classic case.
• URLs generated by software you can’t edit. A third-party app or file store where you control the web server but not the HTML.

For normal HTML pages, the meta tag is usually easier to see and audit. Use whichever one your team can maintain, but avoid setting different values in both.

Supported directives

Google currently supports these values:

Directive	What it does
all	No restrictions (the default; has no effect).
noindex	Don’t show this URL in search results.
nofollow	Don’t follow the links on this page.
none	Same as noindex, nofollow.
nosnippet	Don’t show a text snippet or video preview.
indexifembedded	Allow indexing when embedded in another page with an iframe, even with noindex.
max-snippet: [number]	Limit the text snippet to this many characters.
max-image-preview: [none, standard, large]	Limit the size of image previews.
max-video-preview: [number]	Limit video previews to this many seconds.
notranslate	Don’t offer a translation of this page in results.
noimageindex	Don’t index images on this page.
unavailable_after: [date/time]	Stop showing this URL after the date.

noarchive is now listed by Google as a historical rule with no effect. Header names, user agent names and values aren’t case-sensitive.

Syntax

Combine directives with commas, or send several headers:

X-Robots-Tag: noindex, nofollow

X-Robots-Tag: noimageindex
X-Robots-Tag: unavailable_after: 25 Jun 2027 15:00:00 PST

Target a specific crawler by putting its name first:

X-Robots-Tag: googlebot: nofollow
X-Robots-Tag: otherbot: noindex, nofollow

Without a user agent, the directives apply to every crawler that supports them.

Examples by server

Apache: noindex every PDF

In .htaccess or the virtual host config (requires mod_headers):

<Files ~ "\.pdf$">
  Header set X-Robots-Tag "noindex, nofollow"
</Files>

Apache: noindex an entire staging site

Header always set X-Robots-Tag "noindex, nofollow"

Nginx: noindex every PDF

location ~* \.pdf$ {
    add_header X-Robots-Tag "noindex, nofollow";
}

Nginx: noindex an entire staging site

server {
    server_name staging.example.com;
    add_header X-Robots-Tag "noindex, nofollow" always;
    # ...
}

Two nginx details that catch people out. First, add_header only applies to successful and redirect responses unless you add always. Second, if a location block has its own add_header, it stops inheriting every add_header from the server block, including this one. Repeat the header in those locations or use an include file.

Application code

Express:

app.use('/downloads', (req, res, next) => {
  res.set('X-Robots-Tag', 'noindex');
  next();
});

Flask:

@app.after_request
def noindex_staging(response):
    if app.config.get("STAGING"):
        response.headers["X-Robots-Tag"] = "noindex, nofollow"
    return response

On a CDN, Cloudflare Transform Rules and similar response-header rules can add it without touching the origin.

How to check a URL’s X-Robots-Tag

From the command line:

curl -sI https://example.com/whitepaper.pdf | grep -i x-robots-tag

-I sends a HEAD request, which some servers answer differently from GET. If the result looks wrong, check the real response:

curl -s -D - -o /dev/null https://example.com/whitepaper.pdf | grep -i x-robots-tag

In the browser, open DevTools → Network, click the request and look under Response Headers. For Google’s view, run the URL through URL Inspection in Search Console: a page excluded by the header shows as “Excluded by ‘noindex’ tag”, the same status as the meta tag.

Our free canonical tag checker also shows a page’s X-Robots-Tag next to its canonical, which is where conflicting signals tend to hide.

Common mistakes

Blocking the URL in robots.txt as well. If robots.txt disallows a URL, Google never fetches it and never sees the header, so the noindex has no effect and the URL can still appear in results if other sites link to it. To deindex something, allow crawling and send noindex. Test a path with our robots.txt tester.

Shipping the staging config to production. The most expensive one-line mistake in SEO. A site-wide noindex header on launch day can deindex the whole site within days. Check the headers of a few production URLs after every deploy and infrastructure change, and keep the staging header in environment-specific config, not a shared file.

Setting different values in the header and the meta tag. When directives conflict, Google applies the more restrictive one. A page with index in its meta tag and noindex in its header is noindexed.

Expecting noindex to remove pages instantly. Google has to recrawl the URL to see the header. For urgent removals, use the Removals tool in Search Console as well.

Using nofollow to hide pages. nofollow on a page only affects the links on that page. It doesn’t stop the page itself from being indexed.

Quick answers

Does Bing support X-Robots-Tag? Yes, Bing supports the header for noindex and nofollow, along with other directives.

Is X-Robots-Tag better than the robots meta tag? Neither is stronger. They’re read the same way. The header is necessary for non-HTML files and convenient for site-wide rules.

Can X-Robots-Tag block crawling? No. It controls indexing and how results are shown. Only robots.txt controls crawling.

Can I use X-Robots-Tag for AI crawlers? Some AI crawlers honor it and many don’t. robots.txt is the more widely supported control; see the AI crawler robots.txt guide.

-Sethers