robots.txt and AI crawlers: are you letting GPTBot and ClaudeBot in?

Check the robots.txt file on your website (it is located at yourbusiness.com/robots.txt). If it blocks user agents like GPTBot, ClaudeBot, or Google-Extended, the corresponding AI models cannot read your content at all, no matter how well it is written.

robots.txt is an old, simple standard that tells crawlers which parts of a website they are allowed to visit. It was created long before AI models existed, but is now actively used by several of them to decide whether your content can be pulled in for training or for real-time search. The problem is that many websites, often through default settings in a CMS or an overzealous security plugin, block these crawlers without anyone at the business having made a deliberate choice about it.

Which AI crawlers you should know about

• GPTBot: OpenAI's crawler, used among other things for training data for future ChatGPT models.
• OAI-SearchBot: OpenAI's own crawler for search functionality in ChatGPT.
• ClaudeBot and Claude-Web: Anthropic's crawlers, used for training and for fetching page content when Claude searches the web.
• Google-Extended: controls whether your content can be used for Google's AI features (including Gemini), separate from regular Googlebot, which controls classic search.
• PerplexityBot: Perplexity's crawler for real-time search in their answers.

Note that Google-Extended is separate from Googlebot. That means you can allow regular Google indexing while (knowingly or not) shutting Gemini out of your content. It is one of the most common misconfigurations we see.

Why a business would block them in the first place

There are legitimate reasons: some do not want their own content used to train models they are not paid for, especially media outlets and publishers with a lot of original material. That is a valid trade-off. But for most businesses that want to be recommended by ChatGPT, Claude, or Gemini when someone asks about their industry, the answer is the opposite: you want these crawlers to come in, read the content, and understand who you are.

How to check what is actually allowed

Open yourbusiness.com/robots.txt directly in your browser. Look for lines with "User-agent:" followed by the name of one of the crawlers above, and check whether the following "Disallow:" line blocks the entire website (Disallow: /) or only parts of it. If the file is missing entirely, the default is usually that all crawlers are allowed in, but some hosting platforms and security services block at a level you cannot see in the file itself, for example via a firewall rule. In that case, you also need to check the settings with your hosting provider.

How to safely open up

1. Identify which crawlers are blocked today, either manually or via the AI readiness checker.
2. Remove or adjust the Disallow rules for the crawlers you want to let in. Most CMS platforms let you edit robots.txt directly in the settings.
3. Leave sensitive areas (admin panels, payment pages, internal tools) blocked for all crawlers, that applies regardless.
4. Check again after a few weeks to confirm the change has actually taken effect, since some crawlers cache robots.txt for a period before reading it again.

Letting crawlers in is necessary, but not sufficient. The content also has to actually be worth reading when the crawler arrives, with the right structure and structured data that makes facts explicit. Some websites also choose to publish an llms.txt that points the crawler directly to the most important content.