Skip to content

Light version: pages open quickly, without the 3D graphics. You are seeing the light version because the 3D graphics would be slow on this device. The content is the same.

Turn on 3D

Preparing the graphics. This can take a moment on the first visit.

Henry Lõiv

Web design & development

Quick Answer

Can you block GPTBot while allowing ChatGPT search?

SEO & AI Search 1 min read

Yes. OpenAI provides separate controls for GPTBot and OAI-SearchBot. Blocking GPTBot indicates that content gathered by that crawler should not be used to train foundation models. OAI-SearchBot serves ChatGPT search. 1

An example robots.txt rule

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Merge the relevant rules into your existing file. Do not replace the entire file with this example and accidentally remove sitemap declarations or other necessary crawler restrictions.

Check your firewall and CDN too. Permission in a text file cannot overcome a block elsewhere. A claimed user-agent name is not proof of identity either; OpenAI publishes IP ranges to support verification. 1

What this does not guarantee

Allowing a crawler does not guarantee a citation, ranking or visit. Robots rules are also not access controls: private content needs authentication and authorization. 2

OpenAI separately identifies some user-initiated visits with ChatGPT-User. Automated crawling rules may not apply to those requests in the same way. 1

After changing the file, inspect its public response and relevant server logs. Treat successful crawling as an access check, not evidence that search visibility has already improved. You are setting a content-access preference—not purchasing placement in an AI answer.

Related reading

What llms.txt actually does

Sources

  1. OpenAI: crawler controls
  2. Google: limitations of robots.txt

Fact checked:

Let’s talk about your project

Tell me about your project

Tell me what you’d like to build or improve. You don’t need a finished brief — we’ll work out the scope and next steps together.

01 How can I help?
02 How many pages do you need?
03 Which extras do you need? Choose all that apply
04 When does it need to be ready?
05 Project details
06 Your contact details

Enquiry received

Thank you!

Your enquiry has been received. I usually reply by email within 2 working days.