OAI-SearchBot is the crawler OpenAI uses to surface websites in ChatGPT's search features. If you block it, your pages will not be shown in ChatGPT search answers, though they can still appear as plain links. It is a separate setting from GPTBot, which relates to model training.
What does OAI-SearchBot do?
OpenAI's crawler documentation says the bot is "used to surface websites in search results in ChatGPT's search features." It is how a page becomes eligible to be shown when someone asks ChatGPT a question that triggers a web search. It is not described as a model-training crawler.
Should you allow it?
If you want to be found through ChatGPT, yes. OpenAI recommends allowing it in robots.txt so your site can appear in search results. Sites that opt out "will not be shown in ChatGPT search answers, though can still appear as navigational links."
The decision is simple for most small businesses. A founder who wants new customers to find them has little reason to hide from a search surface. A publisher with licensing concerns may weigh it differently, but that choice should be made on purpose, not by a leftover rule.
How is it different from GPTBot and ChatGPT-User?
OpenAI documents each setting as independent. You can allow search while opting out of training.
| User agent | What OpenAI says it is for | If you disallow it |
|---|---|---|
| OAI-SearchBot | Surfacing sites in ChatGPT search | Not shown in ChatGPT search answers; may still appear as a link |
| GPTBot | Content that may be used to train foundation models | Signals your content should not be used for training |
| ChatGPT-User | Certain user-initiated actions in ChatGPT and Custom GPTs | robots.txt rules may not apply, since a user starts the visit |
Because the three are separate, a blanket rule that blocks "AI bots" can remove you from search without you meaning to. More on the training bot is in what GPTBot is, and the effect on rankings is covered in does blocking GPTBot hurt SEO.
What should robots.txt say?
To allow search and decline training, use separate groups:
User-agent: OAI-SearchBot Allow: / User-agent: GPTBot Disallow: /
If you want both, allow both. A wider walkthrough, including Perplexity, is in allow ChatGPT and Perplexity in robots.txt, and the general idea is in what an AI crawler is.
How long do changes take?
OpenAI says it can take about 24 hours after a robots.txt update for its systems to adjust for search. Do not retest minutes after editing the file.
How do you confirm it is really OAI-SearchBot?
A user-agent string can be faked. OpenAI publishes IP ranges for the bot at openai.com/searchbot.json. Compare the addresses in your server logs with that list before trusting a hit, and see AI crawler monitoring for how to read logs.
Does allowing it guarantee citations?
No. Access only makes you eligible. Whether a page is chosen depends on how well it answers the question, which is the focus of how to get cited by ChatGPT. Nightcrop's site audit checks that AI crawlers can reach your site, which catches an accidental block quickly.
Frequently Asked Questions
Is OAI-SearchBot the same as GPTBot?
No. OpenAI lists them as separate crawlers with separate purposes, and each robots.txt setting is independent of the other.
Will blocking OAI-SearchBot hurt my Google rankings?
OpenAI's documentation does not tie it to Google, and it is not a Google crawler. The effect is on whether you appear in ChatGPT search answers.
Does ChatGPT-User follow robots.txt?
OpenAI says robots.txt rules may not apply because a user initiates those visits. Use OAI-SearchBot for search opt-outs.
How do I verify a visit came from OpenAI?
Match the requesting IP address against the list OpenAI publishes for the bot. Do not rely on the user-agent text alone.
How soon will a robots.txt change take effect?
OpenAI says up to about 24 hours for search. Wait a day before judging the result.