To monitor AI crawlers, read your server or CDN access logs and filter by user agent for names like GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot and Perplexity-User. Then confirm each hit against the IP lists the companies publish, because a user agent string can be faked. Analytics scripts will not show these visits.
Why can't you see AI crawlers in analytics?
Tools like Google Analytics run JavaScript in a visitor's browser. Most crawlers do not run that script, so they never register. The only place every request is recorded is the server side: your web server logs, your CDN or host's request logs, or a firewall log. If you cannot see a log, ask your host where request logs are kept or whether they can be exported.
Which AI crawlers should you look for?
The same company often runs several bots with different jobs. Treating them as one is the most common mistake. Based on the vendors' own documentation:
| User agent | Company | What it does | robots.txt |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT's search features | Respected. Opted-out sites are not shown in ChatGPT search answers. |
| GPTBot | OpenAI | Crawls content that may be used to train OpenAI's models | Disallowing it signals the content should not be used for training. |
| ChatGPT-User | OpenAI | Visits a page when a user asks ChatGPT to | OpenAI says robots.txt rules may not apply, since a user initiated the action. |
| PerplexityBot | Perplexity | Surfaces and links sites in Perplexity search results | Perplexity recommends allowing it. It says this bot is not used for AI model training. |
| Perplexity-User | Perplexity | Visits a page when a user's question needs it | Perplexity says it generally ignores robots.txt, since a user requested the fetch. |
Sources: OpenAI's crawler documentation and Perplexity's bot guide. Both companies can change these, so recheck them when something in your logs looks new. For what to put in robots.txt, see our robots.txt guide for ChatGPT and Perplexity.
How do you find them in your logs?
A basic access log line contains the time, the requested path, the status code and the user agent. Search for the bot names. On a server where you can use the command line, this counts requests per bot:
grep -c "OAI-SearchBot" access.loggrep -c "PerplexityBot" access.loggrep -c "ChatGPT-User" access.log
If you use a CDN or host dashboard instead, look for a bot or user agent filter and search the same names. Then answer four questions:
- Which bots show up at all? A missing search bot is a signal to check access.
- Which pages do they request? If they only fetch your home page, they may not be finding the rest. Check your internal linking and sitemap.
- What status codes do they get? Mostly 200 is healthy. A run of 403, 429 or 5xx responses means something is blocking or overloading them.
- Does it match your intent? If you blocked GPTBot on purpose, it should stop appearing once the change takes effect. OpenAI notes robots.txt changes can take about 24 hours to be reflected, and Perplexity says up to 24 hours as well.
How do you check that a bot is real?
Anyone can put "GPTBot" in a request's user agent. To tell the real crawler from an imitator, compare the visiting IP address with the vendor's published list.
- OpenAI publishes IP lists per bot, for example
openai.com/gptbot.json,openai.com/searchbot.jsonandopenai.com/chatgpt-user.json. - Perplexity publishes
perplexity.ai/perplexitybot.jsonandperplexity.ai/perplexity-user.json.
Match the IP in your log to the ranges in the matching file. Those files use CIDR ranges, so use a CIDR matching tool rather than reading by eye. For comparison, Google describes a similar process for its own crawler in its guide to verifying Googlebot: check reverse DNS, then forward DNS, or compare against Google's published ranges. Do not block or allow a bot on user agent alone if the decision matters.
What do the results tell you?
Seeing a bot request your pages is not the same as being cited. A visit only shows the content was fetched. Whether it appears in an answer depends on things your logs cannot show. Treat the logs as a check on access, and the answers themselves as the check on visibility. Our guides to getting cited by ChatGPT and getting cited by Perplexity cover what you can influence.
Two readings are worth acting on:
- User-triggered bots visiting. Requests from ChatGPT-User or Perplexity-User mean a person asked an assistant something that led to your page. That is a sign your page is being used to answer a question.
- Search bots absent or blocked. If OAI-SearchBot or PerplexityBot never appear, or only receive errors, fix access first. A firewall or bot-protection rule is a common cause.
How often should you check?
A monthly look is enough for most small sites. Check straight after you change robots.txt, switch hosts, add a firewall rule or turn on bot protection, since those are the changes that silently cut off crawlers.
Frequently Asked Questions
Can Google Analytics show AI crawler visits?
No, not reliably. Crawlers generally do not run the analytics script, so server or CDN logs are the place to look.
What is the difference between GPTBot and OAI-SearchBot?
OpenAI describes GPTBot as the crawler for content that may be used to train its models, and OAI-SearchBot as the one that surfaces sites in ChatGPT's search features. They are controlled separately in robots.txt.
Why does ChatGPT-User ignore my robots.txt?
OpenAI says that because these visits are started by a user, robots.txt rules may not apply. Perplexity says the same of Perplexity-User. Use access controls on your server if you need to restrict them.
How do I know a crawler is not fake?
Compare the request's IP address with the vendor's published IP list for that bot. A matching user agent with a non-matching IP is not the real crawler.
Should I block AI crawlers?
That depends on your goals. Blocking search bots can keep you out of the answers they power, while blocking a training bot is a separate choice. Decide per bot, not for all of them at once.