GPTBot is OpenAI's web crawler for collecting content that may be used to train its AI models. It is separate from OAI-SearchBot, which powers ChatGPT search results. Blocking GPTBot opts your content out of training, but it does not by itself remove you from ChatGPT search answers.
What does GPTBot do?
OpenAI documents three bots on its crawler page, each with a different job. GPTBot "is used to crawl content that may be used in training our generative AI foundation models." It identifies itself in server logs with a user-agent string containing GPTBot.
It does not visit your site when a person asks ChatGPT a question. That is handled by the other two bots below.
How is GPTBot different from OAI-SearchBot and ChatGPT-User?
| Bot | Purpose | Follows robots.txt? |
|---|---|---|
| GPTBot | Collects content that may be used for model training | Yes |
| OAI-SearchBot | Surfaces sites in ChatGPT's search features | Yes |
| ChatGPT-User | Visits a page when a user's action in ChatGPT asks for it | Not necessarily, because the visit is user-initiated |
The table follows OpenAI's own description. Two points are easy to miss. First, OpenAI says sites that opt out of OAI-SearchBot "will not be shown in ChatGPT search answers," so that is the bot to allow if you want to be found. Second, for ChatGPT-User, OpenAI says robots.txt rules may not apply, so you cannot rely on robots.txt to control it.
Should you block GPTBot?
It depends on what you want from your content. There is no single right answer, so decide on the trade-off.
- Block it if you do not want your pages used for model training, for example because the content is your product (paid research, a members area, original data).
- Allow it if you are happy for your public articles to be part of the training pool and want the widest reach. Allowing GPTBot is not a guarantee of being cited, and nobody outside OpenAI can promise a training effect.
For most small businesses publishing marketing content, the more important decision is to keep OAI-SearchBot allowed. That is the bot tied to ChatGPT search visibility. Our guide to getting cited by ChatGPT covers what else you control.
How do you block or allow GPTBot in robots.txt?
Robots.txt is a plain text file at the root of your site, for example yoursite.com/robots.txt. It tells crawlers which URLs they may request. To allow search but opt out of training, add:
User-agent: OAI-SearchBot Allow: / User-agent: GPTBot Disallow: /
To allow both, remove the GPTBot rule or replace Disallow: / with Allow: /. For a step-by-step version covering more bots, see how to allow ChatGPT and Perplexity in robots.txt.
OpenAI says it can take about 24 hours after a robots.txt update for its systems to adjust for search results, so do not expect an instant change.
Does blocking GPTBot affect Google rankings?
No direct effect is documented. Google uses its own crawlers, and GPTBot is not one of them. A separate point is that robots.txt is for managing crawler traffic. Google says it "is not a mechanism for keeping a web page out of Google," and it recommends noindex or password protection for that. If you want to understand the difference, read robots.txt vs noindex.
Also note that Google's reference is about Google Search. Other companies have their own crawlers and their own rules, so each needs its own line in robots.txt if you want to control it.
How can you tell whether GPTBot visits your site?
Check your server or CDN logs for requests whose user-agent contains GPTBot. Keep in mind that a user-agent string can be faked by anyone, so a log line alone does not prove the visit came from OpenAI. OpenAI's page says it publishes details for its bots, and you can compare against that before trusting a line. Our post on AI crawler monitoring walks through what to look for.
Nightcrop's site audit checks that AI crawlers can actually reach your site, which is a quick way to catch an accidental block.
What are the common mistakes?
- Blocking everything with one rule. A blanket
User-agent: *withDisallow: /blocks search bots as well as training bots. - Treating GPTBot as the ChatGPT search bot. They are different bots with different purposes.
- Assuming robots.txt hides a page. A blocked URL can still be known to others through links. Use noindex or a login for private pages.
- Forgetting to retest. After any change, open your robots.txt in a browser and read it as the crawler would.
Frequently Asked Questions
Is GPTBot the same as ChatGPT?
No. GPTBot is a crawler that gathers content for possible model training. ChatGPT is the product people use, and it relies on other bots for search and user requests.
Will blocking GPTBot remove me from ChatGPT answers?
Not on its own. OpenAI ties search visibility to OAI-SearchBot, so blocking GPTBot alone leaves that setting unchanged.
Does GPTBot obey robots.txt?
OpenAI says it does. A rule for the GPTBot user agent is the documented way to opt out.
Does ChatGPT-User obey robots.txt?
OpenAI says robots.txt rules may not apply, because those visits happen when a user asks for them. Plan on that bot being able to fetch pages you have publicly exposed.
Do I need an llms.txt file instead?
No. It is a different tool and not a replacement for crawler rules. See do you need an llms.txt file for what it does and does not do.