Free tool
Decide which crawlers can read your site, allow the AI bots so you show up in their answers, or block the ones you don't want, then copy a correct robots.txt file ready to upload.
Save this as a plain text file named robots.txt and upload it to the root of your site, so it lives at yourdomain.com/robots.txt.
Good to know
It's a simple text file at the root of your website that tells crawlers which parts of your site they can read. It was built for search engines and now also controls whether AI tools can use your content.
When people ask ChatGPT or Perplexity for a recommendation, they can only mention you if they're allowed to read your site. For most local businesses, allowing them is the smart choice.
Google-Extended only controls AI training. It's separate from normal Googlebot, so blocking AI crawlers has no effect on where you rank in ordinary Google search.
Reputable crawlers respect robots.txt, but it can't physically stop a bad actor. To truly block something, you need server-level rules. For the major AI companies, robots.txt does the job.
Need a hand?
Getting your robots.txt right, so the good bots see you and the ones you don't want are kept out, is part of what I do for every client. If you'd rather not touch the file yourself, I'll sort it.