7-day websites, £60 a month. Only 5 spots, ends in Claim a spot

Free tool

Are AI crawlers blocked from your site?

Find out instantly. Check whether GPTBot, ClaudeBot, PerplexityBot and other AI crawlers can reach your website, read live from your robots.txt file.

As AI tools increasingly read online content, it helps to know how your website is being accessed. Enter your domain to see whether popular AI crawlers can reach your pages, spot any restrictions, and decide which bots you want to allow or block.

We read your site's /robots.txt and check it against 12 known AI crawlers.

Two ways to set it up

Allow them, or block them

Two complete robots.txt examples you can copy and paste. One lets AI crawlers in, one keeps them out. Here's what each one means for you.

Recommended for businesses

Allow AI crawlers in

Let AI tools read your site so your business can show up in their answers.

robots.txt
User-agent: *
Allow: /

Pros

  • You can appear when people ask ChatGPT, Gemini or Perplexity for a recommendation.
  • A new source of customers that most competitors aren't using yet.
  • Zero effect on your normal Google rankings.
  • Free, and it's the default, nothing to install.

Cons

  • Your public content may be used to help train AI models.
  • You give up some control over how your words and images are reused.

Best for: local businesses, tradespeople, shops and service providers, anyone who wants customers to find them.

For creators & publishers

Block AI crawlers

Stop AI tools reading your site so your original work isn't used without permission.

robots.txt
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-Web
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /

Pros

  • Your writing, images and ideas aren't used to train AI models.
  • Full control over how your original work is used.
  • Sensible if your content is your actual product.

Cons

  • You won't appear in AI search results or recommendations.
  • You lose a fast-growing source of visibility and customers.

Best for: artists, photographers, writers, designers and publishers whose original content is the product.

In short: if you want customers to find you, allow them. If your original work is your livelihood and you'd rather protect it, block them. You can change your mind any time, just update the file.

Good to know

Questions, answered

A plain text file at the root of your site (yourdomain.com/robots.txt) that tells crawlers which parts of your site they may or may not access. It was built for search engines and now also controls AI bots like GPTBot and ClaudeBot.
Because they do different jobs. OpenAI alone runs GPTBot (trains ChatGPT), OAI-SearchBot (lists you in ChatGPT Search) and ChatGPT-User (opens your page when a user asks about it). That split is useful: you can block training while staying visible in AI search.
No. Google-Extended only controls whether your content trains Gemini. It is completely separate from Googlebot, which handles normal search. Blocking AI crawlers does not affect your search rankings.
Always at the root: yourdomain.com/robots.txt. If nothing loads there, you don't have one yet. Create a plain text file called robots.txt and upload it to your site's root folder.
The major, reputable ones (OpenAI, Anthropic, Google, Perplexity and Apple) publicly commit to respecting it. It is a request, not a wall, so a small number of bad actors ignore it. Blocking those for certain requires server-level rules rather than robots.txt.

Need a hand?

Not sure what your results mean?

Whether you want AI crawlers in or out, we'll set your robots.txt up properly and make sure the right bots can see your site. It's part of what we do for every client.

Get in touch See our SEO service