PerplexityBot is Perplexity's crawler. It finds and links sites in Perplexity search results, and Perplexity says it is not used to crawl content for AI foundation models. A second agent, Perplexity-User, fetches a page when a person asks a question. The two follow different robots.txt rules.
What is PerplexityBot?
PerplexityBot is the crawler that builds Perplexity's search index. Perplexity's bot documentation says it is designed to "surface and link websites in search results" and is not used to crawl content for AI foundation models. Its user agent string is:
Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
Perplexity recommends that sites allow it in robots.txt if they want to appear in its results. That makes it closer to a search crawler than to a training crawler, which is the opposite of GPTBot, where blocking is a training opt-out.
What is Perplexity-User?
Perplexity-User is a second agent. According to the same documentation, it supports user actions: when someone asks Perplexity a question, it may visit a page to help answer and then link to that page in the response. Its user agent is:
Mozilla/5.0 (compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)
Perplexity also states it is not used for web crawling or for collecting training content. The important difference is robots.txt: the documentation says this fetcher generally ignores robots.txt rules, because a user asked for the page. A person's request, not a scheduled crawl, is what triggers the visit.
PerplexityBot vs Perplexity-User at a glance
| PerplexityBot | Perplexity-User | |
|---|---|---|
| Triggered by | Scheduled crawling | A person's question |
| Purpose | Surface and link sites in search results | Fetch a page to help answer a question |
| Follows robots.txt | Yes, Perplexity recommends allowing it | Generally ignores it |
| Used for foundation model training | No, per Perplexity | No, per Perplexity |
So robots.txt is a lever for the first agent only. If you want to be absent from Perplexity entirely, a robots.txt rule will not do it for the second.
Should you allow PerplexityBot?
If you want your pages cited in Perplexity answers, yes. A page the crawler cannot reach is unlikely to be linked. If you have a reason to keep content out of AI products, blocking it is your choice, but understand the trade: you give up listing in that engine. We cover the options in how to allow ChatGPT and Perplexity in robots.txt.
An allow rule is short. It names the agent and permits the whole site:
User-agent: PerplexityBotAllow: /
Perplexity notes that changes can take up to 24 hours to be reflected in its systems, so do not judge the result an hour after editing.
Why might PerplexityBot still be blocked?
This is the gap most guides skip. A correct robots.txt is not the only thing between a crawler and your page. Firewalls and bot filters run first, and they can turn the crawler away even when robots.txt says yes.
- Check both the user agent and the IP. Perplexity advises combining user agent matching with IP address verification, and publishes official JSON lists of IP ranges for each agent. Anyone can type a user agent string, so the IP check is what proves the visitor is real.
- Keep the IP lists fresh. The ranges change, so automate updates instead of pasting them once.
- Know that Cloudflare's Bot Fight Mode cannot be skipped by a custom rule. Cloudflare's documentation states that you cannot bypass or skip Bot Fight Mode using WAF custom rules or Page Rules. It recommends Super Bot Fight Mode, which supports skip rules, for exceptions. Perplexity's guide repeats the same limitation.
- Order your rules. An allow rule must sit above any rule that blocks.
If you use Cloudflare or another firewall and PerplexityBot never shows up, this is the first place to look, before you rewrite robots.txt.
How do you tell whether it is visiting?
Search your server or CDN logs for the strings PerplexityBot and Perplexity-User, then confirm the IP addresses against Perplexity's published lists. A visit from the first means you are in its crawl. A visit from the second means someone asked a question that led to your page. Our guide to AI crawler monitoring walks through a log review, and the general idea behind these agents is in what is an AI crawler.
A crawl visit is not a citation, though. To see whether your pages are actually quoted in answers, you have to ask the questions your customers ask and look at the sources. Nightcrop tracks whether AI answer engines cite your domain alongside your organic rankings. For the content side, read how to get cited by Perplexity.
Frequently Asked Questions
Does PerplexityBot train AI models?
Perplexity says it does not. Its documentation states PerplexityBot is not used to crawl content for AI foundation models, and it makes the same statement for Perplexity-User.
Does Perplexity-User obey robots.txt?
Generally no. Perplexity says this fetcher ignores robots.txt rules because a user requested the page. Use your firewall if you need to restrict it.
How long does a robots.txt change take to apply?
Perplexity says it may take up to 24 hours for its systems to reflect changes. Wait a full day before you decide whether an edit worked.
Why does my site not appear in Perplexity?
Check that robots.txt allows PerplexityBot, then check your firewall. A bot filter can block the crawler even when robots.txt allows it. If both are clear, the page may simply not be a strong answer yet.
Is blocking PerplexityBot bad for Google SEO?
Perplexity's crawler is separate from Googlebot, so blocking it does not change Google crawling. You would only lose potential listing in Perplexity.