Guides · 2 October 2026
Is your robots.txt blocking AI crawlers?
Short answerOpen yoursite.co.uk/robots.txt in a browser. If you see a line saying Disallow: / under User-agent: *, you are telling every search and AI crawler to stay away. A safe default for a local business is to allow crawling and list your sitemap.
The two-minute check
- Type your website address followed by /robots.txt.
- Look for
Disallow: /. On its own underUser-agent: *it blocks everything. - Also check your page source for
<meta name="robots" content="noindex">. Some site builders and "coming soon" modes add it, and it hides the page from search.
Crawler names you may see
- Googlebot: Google Search. Google AI features rest on the same index.
- Bingbot: Bing, which also feeds Copilot.
- OAI-SearchBot: OpenAI's search crawler. GPTBot is its training crawler.
- PerplexityBot and ClaudeBot: Perplexity's and Anthropic's crawlers.
- Google-Extended: a control for whether Google uses your content for Gemini training, separate from Search.
A safe starting point
User-agent: *
Allow: /
Sitemap: https://yoursite.co.uk/sitemap.xmlWhether to allow AI training crawlers is your choice. Blocking search crawlers is what makes you invisible.
Questions people ask
Does blocking GPTBot hide me from ChatGPT search?
OpenAI describes them as separate controls: OAI-SearchBot is for search and GPTBot for training. Check OpenAI's current documentation before changing either.
Related guides
- Why isn't my business showing up on ChatGPT?
- How to make your website show up on ChatGPT
- Do I need an llms.txt file?
Sources
Want to know how AI presents your business? Our team prepares a free personal report for you, with a score out of 100, within 2 working days.
Check my business for free