An AI crawler is an automated program that requests web pages for an AI company. Some collect content for model training, some find pages for AI search results, and some fetch a page only when a user asks. Telling these three jobs apart is what lets you control access without disappearing from AI answers.
What do AI crawlers actually do?
A crawler is software that visits URLs and reads what it finds. Search engines have done this for decades. AI companies run their own, and a single company often runs several with different purposes. That is why "should I block AI bots?" has no single answer. It depends on which bot and which job.
What are the three types?
| Type | Job | Examples | Blocking it means |
|---|---|---|---|
| Training crawler | Collects content that may be used to train models | GPTBot | Your content is opted out of training |
| Search crawler | Finds pages to show in AI search results | OAI-SearchBot, PerplexityBot | You may not be shown in those answers |
| User-triggered fetcher | Loads one page because a person asked | ChatGPT-User, Perplexity-User | Often cannot be blocked with robots.txt |
The categories mirror how Google describes its own crawlers: common crawlers, special-case crawlers and user-triggered fetchers, per its crawler overview. Google says its common crawlers always respect robots.txt for automatic crawls.
How do OpenAI's bots work?
OpenAI documents its bots on its bots page. It says disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models. It says OAI-SearchBot surfaces websites in search results in ChatGPT's search features, and that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. For ChatGPT-User, it says actions are initiated by a user, so robots.txt rules may not apply.
For more detail on the first of these, read what GPTBot is and whether to block it.
How do Perplexity's bots work?
Perplexity's bot guide describes two. PerplexityBot surfaces and links websites in Perplexity search results, and Perplexity says it is not used to crawl content for AI foundation models. Perplexity-User supports user actions, and because a user requested the fetch, it generally ignores robots.txt rules. Perplexity recommends allowing PerplexityBot in robots.txt if you want to appear in its results.
How do you control AI crawlers?
- Decide what you want. Most sites that publish marketing content want to be found in AI answers. Some sites sell the content itself and want to opt out of training.
- Write rules per bot. Each bot has its own user-agent name. A rule for one does not affect another.
- Avoid blanket blocks. A
User-agent: *withDisallow: /blocks search bots too. - Retest. Open
yoursite.com/robots.txtand read it. Both OpenAI and Perplexity say changes can take up to about 24 hours to take effect.
A common "find me, don't train on me" setup looks like this:
User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: GPTBot Disallow: /
The full walkthrough is in how to allow ChatGPT and Perplexity in robots.txt.
What can robots.txt not do?
Robots.txt is a request, not a lock. Well-behaved bots follow it. User-triggered fetchers may not. And it only covers crawling: it does not remove a page that is already known elsewhere. For pages that must stay private, use a login. If you are unsure how robots.txt differs from noindex, see robots.txt vs noindex.
How do you know which AI crawlers visit you?
Look at your server or CDN logs and search for the bot names above. Remember that a user-agent string can be faked, so a name alone does not prove who sent the request. Our guide to AI crawler monitoring covers how to read the logs. To check whether a visit gets cited anywhere, see how to track AI citations.
Nightcrop's homepage offers a free SEO and AEO visibility audit if you want a quick check of how your site looks to search and AI systems.
Frequently Asked Questions
Is an AI crawler the same as Googlebot?
No. Googlebot is Google's crawler for Search. AI companies run their own crawlers under their own names.
Should I block all AI crawlers?
Probably not if you want to be found. Blocking search bots such as OAI-SearchBot can remove you from ChatGPT search answers, per OpenAI.
Do AI crawlers respect robots.txt?
The training and search bots from OpenAI and Perplexity document that they do. User-triggered fetchers may not.
Does allowing GPTBot guarantee I get cited?
No. GPTBot relates to training, not to search citations. OAI-SearchBot is the bot tied to ChatGPT search.
How do I tell a real AI crawler from a fake one?
Compare the request against the details the company publishes. A user-agent string alone is easy to fake.