GEO · Technical AI crawlers and robots.txt: how to set up GPTBot, OAI-SearchBot and others correctly

How AI systems read websites

There are three kinds of access by AI systems. Training crawlers collect text used to train future models. Search crawlers build their own index, from which the system selects and links sources when a question comes in. Fetches on behalf of a user happen when someone asks the chat to open a specific page or the AI checks something live for an answer.

For your company’s visibility, the search crawlers matter most. If you block OAI-SearchBot, OpenAI says your website will not appear in ChatGPT’s search answers. A blocked training crawler, on the other hand, only means your content should not be used in future training data.

The most important AI crawlers at a glance

Two details are worth knowing. OpenAI and Perplexity point out that their fetches on behalf of users do not always follow robots.txt, because a person requested the page. And according to Google, Google-Extended has no effect on whether your pages appear in Google Search or how they rank there. AI Overviews and AI Mode are part of Search and are controlled through the normal Googlebot.

Three typical setups

Allow everything

For most businesses this is the right choice: all crawlers may read, except areas with no value for visitors.

User-agent: *
Disallow: /basket/
Disallow: /internal/
Sitemap: https://www.example.com/sitemap.xml

Visible in AI search, but no training

If you do not want your text in training data, block only the training crawlers and control tokens. The search crawlers fall under the general rule and stay allowed.

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xml

What you should weigh up

Whether you block training is a business decision. A block only affects future training runs, knowledge already learned remains. For many small and medium-sized businesses it is rather an advantage if models know the company name and services even without a web search. Publishers and providers of paid content often decide differently.

What EU law says

The EU copyright directive allows automated analysis of text and data unless the rightholder has reserved this in a machine-readable way for content published online (Art. 4(3) of Directive 2019/790). The AI Act requires providers of general-purpose AI models to identify and comply with such reservations (Art. 53(1)(c)). robots.txt is the most widely used machine-readable means for this. Whether it is always sufficient as an effective reservation has not yet been conclusively settled in law.

Common mistakes

• Blanket block left over from a relaunch: Disallow: / for all crawlers often comes from a test environment and stays in place after go-live.
• Bot protection blocks them too: some firewalls, hosting providers and bot protection services block AI crawlers on their own, even if robots.txt allows them.
• Forgotten subdomains: every subdomain such as shop.example.com needs its own robots.txt.
• Wrong names: upper and lower case do not matter, typos do. A “GPT-Bot” blocks nobody.
• Hiding secrets with robots.txt: the file is public. Listing internal paths there makes them easier to find, not harder.

How to check your setup

1. Open https://your-domain/robots.txt in a browser and read each group.
2. Check subdomains and old domains that are still reachable.
3. Use the robots.txt report in Google Search Console.
4. Search your server logs for OAI-SearchBot, Claude-SearchBot and PerplexityBot. The providers publish their IP ranges so that genuine requests can be told apart from fake ones.
5. Ask your hosting or CDN provider whether a bot protection service blocks AI crawlers.

Changes take a little time: OpenAI and Perplexity mention up to 24 hours before their systems take a new robots.txt into account. Whether AI systems then actually name your business is something we measure as a GEO agency in the free check.