AI Search
AI Crawler Rules on Calgary Websites: A 1,025-Site Study
A Calgary study of robots.txt rules for GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, and ChatGPT-User.
Most Calgary websites had no crawler-specific rule
The audit could read AI-crawler robots rules for 1,025 websites. About one in seven had an explicit rule for GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, or CCBot. Crawler-specific rules were much less common for ChatGPT-User and PerplexityBot.
An explicit rule can allow or restrict a crawler. The audit therefore kept the presence of a rule separate from whether homepage access was restricted.
| Crawler | Explicit rule | Homepage restricted |
|---|---|---|
| GPTBot | 15.3% | 3.0% |
| ClaudeBot | 15.0% | 2.8% |
| Google-Extended | 15.0% | 2.8% |
| CCBot | 14.6% | 3.2% |
| Applebot-Extended | 13.9% | 2.9% |
| PerplexityBot | 1.6% | 0.0% |
| ChatGPT-User | 1.5% | 0.0% |
No explicit rule does not mean the business made a choice
When a robots file has no crawler-specific rule, the crawler may fall back to broader instructions. That absence should not be described as a deliberate decision to allow or block AI-assisted search tools.
For many small businesses, the robots file was created by a platform, plugin, or developer and may never have been reviewed as a business policy.
Different crawlers serve different purposes
Crawler names can look similar while supporting different products and uses. A training-focused crawler, a user-requested browsing tool, and a search feature are not automatically the same policy decision.
That is one reason a copied blocklist can create unintended results. Businesses should understand which crawler a rule targets and what outcome they actually want before changing access.
Robots rules are only one part of AI-readability
Allowing a crawler does not make a website clear, useful, or likely to be cited. The site still needs accurate service pages, local context, visible business details, schema, strong internal links, and information customers can trust.
Blocking or allowing access also does not guarantee how any search tool will use or present the content. The practical website work remains clarity, structure, technical health, and useful answers.
Review policy changes carefully
Before changing robots.txt, confirm the current file, identify who manages it, and check whether the website platform or SEO plugin may overwrite manual edits. Keep ordinary search crawlers separate from AI-specific decisions.
After a change, test the exact file being served and keep a short record of why the rule was added. A website update should not accidentally block normal search visibility or important site resources.
A practical starting point for local businesses
Decide what the business wants first: ordinary search visibility, clear AI-assisted search access, content protection, or a more selective policy. Then review the current robots file and website structure together.
The crawler policy should support a broader website plan. It is not a substitute for clear services, trustworthy details, fast pages, and an easy path to contact.
About the crawler-policy data
The study uses one latest compatible audit per hostname from a July 2026 Calgary website archive. Only sites with a measurable robots policy are included in the 1,025-site denominator.
The audit recorded normalized rule presence and homepage access for seven named crawlers. It did not store raw site content or infer business intent from an absent rule.