PerplexityBot and Perplexity-User: How Perplexity Crawls, Fetches and Cites

Perplexity uses two user agents. PerplexityBot crawls the web to build the index Perplexity searches and links to, and Perplexity says it is not used to train foundation models. Perplexity-User fetches a page live when a person’s question needs it, and Perplexity says that fetcher “generally ignores robots.txt rules.” Allow PerplexityBot if you want to be cited; Perplexity-User will usually reach you either way.

What do PerplexityBot and Perplexity-User do?

Perplexity documents both at docs.perplexity.ai/guides/bots (read on 31 October 2026):

	PerplexityBot	Perplexity-User
User agent contains	PerplexityBot/1.0; +https://perplexity.ai/perplexitybot	Perplexity-User/1.0; +https://perplexity.ai/perplexity-user
Job	Surfaces and links websites in Perplexity search results	Visits a page when a user asks a question, to answer it and link to the page
Used for model training	No, per Perplexity	No, per Perplexity
Obeys robots.txt	Yes; changes can take up to 24 hours	“generally ignores robots.txt rules”
IP list	perplexity.ai/perplexitybot.json	perplexity.ai/perplexity-user.json

Both full strings start with Mozilla/5.0 (compatible; ...) and the version number can change, so match the token.

Perplexity’s own advice is direct: “we recommend allowing PerplexityBot in your site’s robots.txt file.” There is no separate training crawler to block, because Perplexity does not say it runs one.

Why does Perplexity-User ignore robots.txt?

Perplexity’s reasoning is the same as OpenAI’s for ChatGPT-User: the request came from a person, so it behaves more like a browser than a crawler. The documentation says “Since a user requested the fetch, this fetcher generally ignores robots.txt rules.”

That has two practical consequences.

1. A robots.txt rule for Perplexity-User is mostly decorative. If you need to stop it, use a firewall or CDN rule that matches both the user agent and the IP ranges Perplexity publishes. Perplexity’s docs recommend matching both, and refreshing the ranges from the JSON regularly.
2. Perplexity-User requests are a good proxy for citations. Because each one starts with a person’s question, a Perplexity-User hit on a URL means somebody’s answer drew on that page. How to see which pages ChatGPT and Perplexity cite shows how to pull those out of your logs.

How often does Perplexity visit?

From this site’s nginx logs (blackseoanalyzer.com):

User agent	May 2026	Sep 2026	Oct 2026
PerplexityBot	191	1,736	1,182
Perplexity-User	not in May’s top nine	994	866

May’s figures only cover the nine busiest AI agents that month, and Perplexity-User was not one of them. PerplexityBot crawled nine times as often in September as in May, then eased off in October. Perplexity-User stayed close to a thousand fetches a month. For comparison, ChatGPT-User made 1,012 fetches in October, so Perplexity’s live fetching is in the same range as ChatGPT’s on this site, despite the difference in their audience sizes.

The human side is small. Perplexity sent 12 human visitors in twelve months, against 354 unique humans from Google in October alone. The full table of every AI crawler is in which AI crawlers hit a website in 2026.

How does Perplexity pick the sources it cites?

Perplexity does not publish a ranking formula, so anyone who tells you exactly how it chooses is guessing. What its documentation does establish is the plumbing: PerplexityBot builds an index, Perplexity-User fetches pages live, and answers link to the pages they used. Everything else here comes from watching what gets fetched.

On this site, the pages fetched on behalf of users by ChatGPT-User, Perplexity-User and Claude-User in the last 90 days were the homepage (1,537 times), then the CLI documentation (28) and one older blog post comparing tools (25). Those pages have three things in common: they answer a specific question near the top, they are plain server-rendered HTML, and they use tables and lists that can be quoted without rewriting. How to write a page an LLM will quote goes through that structure in detail.

The rest of the user-initiated fetch list was scanner noise: requests for environment files and paths like /fetch and /proxy. Somebody is using assistant fetchers to probe sites for secrets, so check your logs for the same pattern and make sure those paths return nothing useful.

What robots.txt rules should you use for Perplexity?

To be cited:

User-agent: PerplexityBot
Allow: /

To stay out of Perplexity search (Perplexity-User may still fetch pages users ask about):

User-agent: PerplexityBot
Disallow: /

If you have a blanket rule like User-agent: * with Disallow: / and only allow named bots, add PerplexityBot to the allow list explicitly. A missing group means PerplexityBot follows your * rules.

How do you verify Perplexity traffic?

# Perplexity requests per token
grep -Eo 'PerplexityBot|Perplexity-User' access.log | sort | uniq -c

# the URLs Perplexity-User fetched most
grep 'Perplexity-User' access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Then compare the source IPs with perplexitybot.json and perplexity-user.json. Requests claiming the token from other addresses are someone else.

What should you check after allowing PerplexityBot?

Access is necessary, not sufficient. PerplexityBot reads the HTML your server returns, so if your content only appears after JavaScript runs, there is little for it to index; can AI crawlers see JavaScript content covers the test. And make sure the rule you meant to write is the rule you wrote: the AI crawler robots.txt guide lists the common copy-paste mistakes.

bseoa’s AI-readiness checks read robots.txt for both Perplexity tokens. A blocked PerplexityBot is reported as Let AI Search Crawlers Read Your Site So It Can Be Cited, and a blocked Perplexity-User as Allow User-Initiated Fetches So Shared Links Work. For a quick one-off test of a single rule, use the free robots.txt tester.

How all of this compares with classic search is on GEO vs SEO.

-Sethers