AI Crawlers, robots.txt and llms.txt: How to Let GPTBot, ClaudeBot, PerplexityBot and Others Read Your Website

Four kinds of bot

Search engine crawlers build traditional indexes: Googlebot, Bingbot, Applebot, Baiduspider, PetalBot and others. Block these and you disappear from search.

AI search crawlers index pages for AI search products: OAI-SearchBot for ChatGPT search, Claude-SearchBot for Claude, PerplexityBot for Perplexity. Allowing them makes you eligible to be cited.

User-triggered fetchers visit a page when a person asks an assistant about it: ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User, Meta-ExternalFetcher. They act on behalf of a real user.

Training crawlers collect content to train models: GPTBot, ClaudeBot, Google-Extended (a control token rather than a separate crawler), Applebot-Extended, CCBot from Common Crawl, Bytespider, meta-externalagent, Amazonbot and others. Whether to allow them is a business choice; for an agency that wants to be known, we allow them.

The vendors document their own bots: OpenAI and Google both publish lists.

The robots.txt we use

Our own robots.txt starts with a wildcard group that allows every bot, adds a Content-Signal line stating that our content may be used for search, AI input and AI training, then names more than forty search and AI crawlers explicitly with Allow rules, and lists both sitemaps. The wildcard alone would be enough; naming each bot removes any doubt for crawlers that look for their own token. If you want to block training but stay in AI search, you would disallow GPTBot, ClaudeBot and Google-Extended while allowing OAI-SearchBot, Claude-SearchBot and PerplexityBot.

The trap: your CDN may block them anyway

Robots.txt is a request; a CDN or firewall can override it. Cloudflare, for example, offers settings that block known AI crawlers and can manage robots.txt on your behalf, and some hosting firewalls treat unfamiliar bots as attacks. If AI tools never cite you, check these settings first. In Cloudflare, look at the AI crawler controls under Security and Bots, and at whether a managed robots.txt is being served instead of yours. Then request your own robots.txt and a few pages with different bot user agents to see whether they return 200.

Server logs are the final word. Most hosting panels let you download access logs; search them for GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot. If those names never appear, or appear only with 403 responses, something between the internet and your site is turning them away, whatever robots.txt says.

What llms.txt is, and is not

llms.txt is a proposed convention, documented at llmstxt.org: a Markdown file at the root of your site that tells language models what the site is and links to its most useful pages. Ours at /llms.txt covers our positioning, prices, contact details, every service, every market page, our home service lead generation pages and every article. It is generated from the same data as the site, so it never goes out of date.

It is not a Google ranking factor, and not every AI company reads it yet. It is cheap to add and helps tools that do read it get your facts right. Keep it factual and short at the top: who you are, what you sell, prices, locations, contact. Update it whenever prices, services or locations change, because a stale llms.txt teaches models the wrong facts.

Structured data and sitemaps still do the heavy lifting

Crawlers find pages through links and sitemaps and understand them through clear HTML and structured data. Keep your XML sitemap complete and current, submit it to Google Search Console and Bing Webmaster Tools, and use IndexNow to tell Bing and others when pages change. Make sure the content that answers questions is in the server-rendered HTML, not loaded later by scripts that some crawlers do not run. See our schema guide.

A ten-minute check for your site

1. Open yourdomain.com/robots.txt and look for any Disallow lines under AI bot names or the wildcard. 2. Check your CDN and firewall bot settings. 3. Confirm your sitemap is listed and loads. 4. Search your brand in ChatGPT, Perplexity and Google AI Mode and note what they say. 5. Look in analytics for referrals from AI tools. If any step surprises you, that is where to start. Our Search Everywhere Optimisation service sets all of this up for clients in the USA, India, the UK, the UAE and Canada.

How we set it up on adsninza.com

Our robots.txt, sitemap.xml, sitemap.txt and llms.txt are all generated by the application from the same data that builds the pages, so a new service, location page or article appears in every file the moment it is published. Every page carries JSON-LD: Organization and WebSite everywhere, then Service with offers, FAQPage, BreadcrumbList and WebPage with speakable sections on service and location pages, and BlogPosting with author, dates and place entities on articles. Content is rendered on the server, so crawlers that do not run JavaScript still see every word. After each publish we notify Bing, Yandex and others through IndexNow and resubmit the sitemap to Google and Bing.

None of this is exotic. It is the same foundation we build for clients, including the October 2026 launches for Black Label Mobile Detailing and Neal K. Prasad, both of which were ranking at the top of Google from their second day, as the clients reported.

A note on privacy and choice

Not every business should allow every bot. Publishers whose content is the product, sites with paywalled or personal data, and regulated businesses may decide differently, and that is legitimate. For most service businesses, though, the content on the website exists to be found and quoted, and blocking the systems that answer buyers' questions works against that goal.

Frequently asked questions