sam@cribbarcreative.co.uk

Free tool

Are AI crawlers blocked from your site?

Find out instantly. Check whether GPTBot, ClaudeBot, PerplexityBot and other AI crawlers can reach your website, read live from your robots.txt file.

As AI tools increasingly read online content, it helps to know how your website is being accessed. Enter your domain to see whether popular AI crawlers can reach your pages, spot any restrictions, and decide which bots you want to allow or block.

We read your site's /robots.txt and check it against 12 known AI crawlers.

Should you block them?

Let AI crawlers in, or keep them out?

There's no single right answer. Here's the honest trade-off, in plain English.

Reasons to allow them

  • Your business can show up when people ask ChatGPT, Gemini or Perplexity to recommend someone.
  • AI search is fast becoming a real source of customers. Block it and you're invisible there.
  • It has zero effect on your normal Google rankings.
  • It's free and needs no effort. Allowing is the default.

Reasons to block them

  • You'd rather your writing, images and ideas weren't used to train AI for free.
  • Your original content is your edge, and you want to protect it.
  • You have premium or members-only material you don't want reproduced in AI answers.
  • You simply want full control over who uses your work.

For most small and local businesses, let them in. Being the answer when someone asks an AI for "a web designer in Newquay" is worth far more than the small downside. If your content is your product, block the training bots but keep the search bots, so you protect your work and still get found. The recipes below do exactly that.

Take control

Copy, paste, done

Pick the goal that fits you, then add the lines to the robots.txt file at the root of your site.

Allow everything

Recommended for most. You appear in AI search and answers.

User-agent: *
Allow: /

Block training, keep AI search

Protect your content but stay findable in AI answers.

# Stops your content training AI models,
# but keeps you visible in AI search
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /

Block every AI crawler

Keep AI out entirely.

User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-Web
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /

Your robots.txt lives at yourdomain.com/robots.txt. No file yet? Create a plain text file named robots.txt in your site's root folder. Changes take effect the next time each crawler visits.

Good to know

Questions, answered

A plain text file at the root of your site (yourdomain.com/robots.txt) that tells crawlers which parts of your site they may or may not access. It was built for search engines and now also controls AI bots like GPTBot and ClaudeBot.
Because they do different jobs. OpenAI alone runs GPTBot (trains ChatGPT), OAI-SearchBot (lists you in ChatGPT Search) and ChatGPT-User (opens your page when a user asks about it). That split is useful: you can block training while staying visible in AI search.
No. Google-Extended only controls whether your content trains Gemini. It is completely separate from Googlebot, which handles normal search. Blocking AI crawlers does not affect your search rankings.
Always at the root: yourdomain.com/robots.txt. If nothing loads there, you don't have one yet. Create a plain text file called robots.txt and upload it to your site's root folder.
The major, reputable ones (OpenAI, Anthropic, Google, Perplexity and Apple) publicly commit to respecting it. It is a request, not a wall, so a small number of bad actors ignore it. Blocking those for certain requires server-level rules rather than robots.txt.

Need a hand?

Not sure what your results mean?

Whether you want AI crawlers in or out, we'll set your robots.txt up properly and make sure the right bots can see your site. It's part of what we do for every client.

Get in touch See our SEO service