---
title: "X-Robots-Tag Header: Syntax, Examples and Common Mistakes"
description: "How the X-Robots-Tag HTTP header works: supported directives, syntax, Apache and Nginx examples, noindexing PDFs and staging sites, and how to check it."
image: "https://www.blackseoanalyzer.com/static/images/black-seo-analyzer-og-image.png"
canonical: "https://www.blackseoanalyzer.com/en/blog/x-robots-tag"
language: "en"
---

# X-Robots-Tag: How to Use the HTTP Header

By [Seth Black](https://www.blackseoanalyzer.com/en/about) · Published September 17, 2026

The X-Robots-Tag is the robots meta tag’s less famous sibling. It does the same job, but in an HTTP response header instead of the HTML. That makes it the only way to control indexing for PDFs, images and other files that don’t have a `<head>`, and the easiest way to noindex an entire staging site from the server config.

## What X-Robots-Tag is

It’s an HTTP response header that tells search engines how to index and display a URL:

```http
HTTP/1.1 200 OK
Content-Type: application/pdf
X-Robots-Tag: noindex, nofollow
```

Any directive that works in a robots meta tag works in the header. These two are equivalent for an HTML page:

```html
<meta name="robots" content="noindex, nofollow">
```

```http
X-Robots-Tag: noindex, nofollow
```

## When to use it instead of the meta tag

- **Non-HTML files.** PDFs, Word documents, images, videos and feeds can’t carry a meta tag. The header is the only way to noindex them.
- **Whole sites or directories.** One line in the server config covers every URL, so you don’t depend on every template getting it right. Staging and preview environments are the classic case.
- **URLs generated by software you can’t edit.** A third-party app or file store where you control the web server but not the HTML.

For normal HTML pages, the meta tag is usually easier to see and audit. Use whichever one your team can maintain, but avoid setting different values in both.

## Supported directives

Google currently supports these values:

| Directive | What it does |
| --- | --- |
| `all` | No restrictions (the default; has no effect). |
| `noindex` | Don’t show this URL in search results. |
| `nofollow` | Don’t follow the links on this page. |
| `none` | Same as `noindex, nofollow`. |
| `nosnippet` | Don’t show a text snippet or video preview. |
| `indexifembedded` | Allow indexing when embedded in another page with an iframe, even with `noindex`. |
| `max-snippet: [number]` | Limit the text snippet to this many characters. |
| `max-image-preview: [none, standard, large]` | Limit the size of image previews. |
| `max-video-preview: [number]` | Limit video previews to this many seconds. |
| `notranslate` | Don’t offer a translation of this page in results. |
| `noimageindex` | Don’t index images on this page. |
| `unavailable_after: [date/time]` | Stop showing this URL after the date. |

`noarchive` is now listed by Google as a historical rule with no effect. Header names, user agent names and values aren’t case-sensitive.

## Syntax

Combine directives with commas, or send several headers:

```http
X-Robots-Tag: noindex, nofollow
```

```http
X-Robots-Tag: noimageindex
X-Robots-Tag: unavailable_after: 25 Jun 2027 15:00:00 PST
```

Target a specific crawler by putting its name first:

```http
X-Robots-Tag: googlebot: nofollow
X-Robots-Tag: otherbot: noindex, nofollow
```

Without a user agent, the directives apply to every crawler that supports them.

## Examples by server

### Apache: noindex every PDF

In `.htaccess` or the virtual host config (requires `mod_headers`):

```apache
<Files ~ "\.pdf$">
  Header set X-Robots-Tag "noindex, nofollow"
</Files>
```

### Apache: noindex an entire staging site

```apache
Header always set X-Robots-Tag "noindex, nofollow"
```

### Nginx: noindex every PDF

```nginx
location ~* \.pdf$ {
    add_header X-Robots-Tag "noindex, nofollow";
}
```

### Nginx: noindex an entire staging site

```nginx
server {
    server_name staging.example.com;
    add_header X-Robots-Tag "noindex, nofollow" always;
    # ...
}
```

Two nginx details that catch people out. First, `add_header` only applies to successful and redirect responses unless you add `always`. Second, if a `location` block has its own `add_header`, it stops inheriting every `add_header` from the `server` block, including this one. Repeat the header in those locations or use an include file.

### Application code

Express:

```js
app.use('/downloads', (req, res, next) => {
  res.set('X-Robots-Tag', 'noindex');
  next();
});
```

Flask:

```python
@app.after_request
def noindex_staging(response):
    if app.config.get("STAGING"):
        response.headers["X-Robots-Tag"] = "noindex, nofollow"
    return response
```

On a CDN, Cloudflare Transform Rules and similar response-header rules can add it without touching the origin.

## How to check a URL’s X-Robots-Tag

From the command line:

```bash
curl -sI https://example.com/whitepaper.pdf | grep -i x-robots-tag
```

`-I` sends a HEAD request, which some servers answer differently from GET. If the result looks wrong, check the real response:

```bash
curl -s -D - -o /dev/null https://example.com/whitepaper.pdf | grep -i x-robots-tag
```

In the browser, open DevTools → Network, click the request and look under Response Headers. For Google’s view, run the URL through **URL Inspection** in Search Console: a page excluded by the header shows as “Excluded by ‘noindex’ tag”, the same status as the meta tag.

Our free [canonical tag checker](https://www.blackseoanalyzer.com/en/tools/canonical-tag-checker) also shows a page’s X-Robots-Tag next to its canonical, which is where conflicting signals tend to hide.

## Common mistakes

**Blocking the URL in robots.txt as well.** If robots.txt disallows a URL, Google never fetches it and never sees the header, so the `noindex` has no effect and the URL can still appear in results if other sites link to it. To deindex something, allow crawling and send `noindex`. Test a path with our [robots.txt tester](https://www.blackseoanalyzer.com/en/tools/robots-txt-tester).

**Shipping the staging config to production.** The most expensive one-line mistake in SEO. A site-wide `noindex` header on launch day can deindex the whole site within days. Check the headers of a few production URLs after every deploy and infrastructure change, and keep the staging header in environment-specific config, not a shared file.

**Setting different values in the header and the meta tag.** When directives conflict, Google applies the more restrictive one. A page with `index` in its meta tag and `noindex` in its header is noindexed.

**Expecting `noindex` to remove pages instantly.** Google has to recrawl the URL to see the header. For urgent removals, use the Removals tool in Search Console as well.

**Using `nofollow` to hide pages.** `nofollow` on a page only affects the links on that page. It doesn’t stop the page itself from being indexed.

## Quick answers

**Does Bing support X-Robots-Tag?** Yes, Bing supports the header for `noindex` and `nofollow`, along with other directives.

**Is X-Robots-Tag better than the robots meta tag?** Neither is stronger. They’re read the same way. The header is necessary for non-HTML files and convenient for site-wide rules.

**Can X-Robots-Tag block crawling?** No. It controls indexing and how results are shown. Only robots.txt controls crawling.

**Can I use X-Robots-Tag for AI crawlers?** Some AI crawlers honor it and many don’t. robots.txt is the more widely supported control; see [the AI crawler robots.txt guide](https://www.blackseoanalyzer.com/en/blog/ai-crawler-robots-txt-guide).

-Sethers

About the author

[Seth Black](https://www.blackseoanalyzer.com/en/about)

Seth is a software engineer and engineering leader who builds bseoa, a Rust-based technical SEO crawler with a GUI and CLI. He writes about the crawling, rendering and indexing problems he runs into on real sites.

[LinkedIn](https://www.linkedin.com/in/seth-black-tx/) · [GitHub](https://github.com/sethblack) · [YouTube](https://www.youtube.com/@SethBlack)

[Back to Blog](https://www.blackseoanalyzer.com/en/blog)

## Discover hundreds of SEO Issues in Seconds

Without Monthly Subscriptions

Comprehensive technical SEO analysis powered by ML and 16 specialized modules. Optional AI-powered insights from Claude, GPT-4, or Gemini. Get actionable insights in seconds, and never pay monthly fees again.

[Download Free Trial](https://www.blackseoanalyzer.com/en/free-trial)

I use AI to generate images for my posts and for general editing, updates, and ironically SEO purposes. I used to draw all of the images for my personal blog (taleas) myself, but as the volume of content I produce has increased, I've turned to AI tools to help create visuals that complement my writing. I go out of my way to generate images that look strange, and don't represent real people. If you ever want to chat about my use of AI, please reach out.

```json
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://www.blackseoanalyzer.com/#organization","name":"Fiscus Technology, LLC","url":"https://www.blackseoanalyzer.com/","logo":{"@type":"ImageObject","url":"https://www.blackseoanalyzer.com/static/images/black-seo-analyzer.png"},"sameAs":["https://sethserver.com/","https://github.com/sethblack"],"founder":{"@id":"https://www.blackseoanalyzer.com/#seth"}},{"@type":"WebSite","@id":"https://www.blackseoanalyzer.com/#website","name":"bseoa","url":"https://www.blackseoanalyzer.com/","publisher":{"@id":"https://www.blackseoanalyzer.com/#organization"}},{"@type":"Person","@id":"https://www.blackseoanalyzer.com/#seth","name":"Seth Black","url":"https://www.blackseoanalyzer.com/about","jobTitle":"Creator of bseoa","sameAs":["https://www.linkedin.com/in/seth-black-tx/","https://github.com/sethblack","https://www.youtube.com/@SethBlack","https://sethserver.com/"]},{"@type":"BlogPosting","@id":"https://www.blackseoanalyzer.com/en/blog/x-robots-tag#article","headline":"X-Robots-Tag: How to Use the HTTP Header","name":"X-Robots-Tag: How to Use the HTTP Header","description":"How the X-Robots-Tag HTTP header works: supported directives, syntax, Apache and Nginx examples, noindexing PDFs and staging sites, and how to check it.","url":"https://www.blackseoanalyzer.com/en/blog/x-robots-tag","datePublished":"2026-09-17T03:21:07","dateModified":"2026-09-17T03:21:07","author":{"@id":"https://www.blackseoanalyzer.com/#seth"},"publisher":{"@id":"https://www.blackseoanalyzer.com/#organization"},"image":{"@type":"ImageObject","url":"https://www.blackseoanalyzer.com/static/images/black-seo-analyzer-og-image.png","width":1200,"height":630},"mainEntityOfPage":{"@type":"WebPage","@id":"https://www.blackseoanalyzer.com/en/blog/x-robots-tag"},"isPartOf":{"@id":"https://www.blackseoanalyzer.com/#website"},"inLanguage":"en-US","isAccessibleForFree":true,"wordCount":1072},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://www.blackseoanalyzer.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://www.blackseoanalyzer.com/en/blog"},{"@type":"ListItem","position":3,"name":"X-Robots-Tag: How to Use the HTTP Header","item":"https://www.blackseoanalyzer.com/en/blog/x-robots-tag"}]}]}
```