Can AI Read Your Real Estate Website? A 30-Minute Crawler Check
AI quick win #2: check robots.txt, your CDN bot settings and whether your key facts are plain text, so ChatGPT, Google and other AI tools can read and cite your site.
Ryan Rockwood · 6 min read
On this page
- Why this matters
- The steps (about 30 minutes)
- 1. Read your robots.txt (5 minutes)
- 2. Check your CDN or firewall (10 minutes)
- 3. Make sure your key facts are text (10 minutes)
- 4. Skip llms.txt (0 minutes)
- What we saw in our market check
- How can I check if it’s working?
- Want us to do it for you?
- Sources
- All ten quick wins
The short version: AI answers can only cite pages their crawlers are allowed to reach and can actually read. Check three things: your robots.txt, your CDN or firewall bot settings, and whether your key facts are on the page as plain text. Thirty minutes, no new software.
- Let the search crawlers in: OAI-SearchBot (ChatGPT search), Googlebot, Bingbot, Claude-SearchBot, PerplexityBot.
- Training crawlers (GPTBot, ClaudeBot, Google-Extended) are a separate choice. Blocking Google-Extended does not remove you from Google’s AI Overviews.
- Your name, brokerage, areas, phone and license number should be readable text, not only an image or a widget.
- You don’t need an llms.txt file. Google says Search ignores it.
Agents ask us: “Is my site even visible to ChatGPT?”, “Should I block AI bots?” and “How do I check without being a developer?” This is quick win #2 of 10. If you ran win #1 and your own site never showed up, start here.
Why this matters
Every AI search tool sends a crawler to read pages. If your site turns that crawler away, you can be the best agent in town and the answer still won’t link to you.
- ChatGPT: OpenAI runs separate bots for separate jobs. OAI-SearchBot is the one that puts sites into ChatGPT search answers; sites that opt out “will not be shown in ChatGPT search answers.” GPTBot is for training, and blocking it says your content shouldn’t be used to train models. ChatGPT-User fetches a page when a person asks it to. OpenAI says robots.txt changes take about 24 hours to apply (OpenAI crawlers).
- Firewalls count too. OpenAI warns that CDNs, firewalls, bot protection, CAPTCHAs and logins can block its crawlers even when robots.txt allows them, and that “a successful HTTP response alone does not confirm that the content loaded” (OpenAI Help Center).
- Google: AI Overviews and AI Mode follow normal Googlebot rules. Google says Google-Extended “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search” (Google’s common crawlers). Google also asks that important content be “available in textual form”, and says you don’t need special AI files or markup (AI features and your website).
- Claude and Perplexity work the same way: Anthropic’s Claude-SearchBot and Perplexity’s PerplexityBot are their search crawlers (Anthropic, Perplexity).
One belief worth correcting: Cloudflare, which sits in front of many websites, used to make headlines for blocking AI bots by default. Its current documentation (updated November 2, 2026) says that for new domains, search, agent and training crawlers are allowed by default. Sites set up earlier keep whatever was set back then, so an older site may still be blocking. Cloudflare also notes its “training” setting covers some crawlers used for both training and search (Cloudflare docs).
The steps (about 30 minutes)
1. Read your robots.txt (5 minutes)
Type yourdomain.com/robots.txt into your browser. You’re looking for Disallow: / under User-agent: * (that blocks everyone) or under any of these names: OAI-SearchBot, Googlebot, Bingbot, Claude-SearchBot, PerplexityBot. If you see one, that crawler is told to stay out of your whole site. A blank file, or one that only blocks admin folders, is fine.
2. Check your CDN or firewall (10 minutes)
If your site uses Cloudflare, open the dashboard’s AI crawler settings and look at the Search and Training policies. If you didn’t set your site up yourself, ask whoever did: “Do we block any AI or search crawlers at the firewall?” Decide on purpose. For visibility, allow search. Training is your call, and it doesn’t affect Google Search.
3. Make sure your key facts are text (10 minutes)
Open your homepage and About page, right-click, and choose “View page source”. Use find (Ctrl+F or Cmd+F) for your name, your brokerage, your phone number, one city you serve and your license number. If any of them only exist inside a logo image, a photo banner or a chat widget, they’re harder for crawlers to read. Ask for them to be added as plain text, for example in the footer.
4. Skip llms.txt (0 minutes)
You’ll hear about adding an “llms.txt” file. Google’s guide says Search ignores these files and that adding one “will neither harm nor help” (Google). We found no statement from OpenAI, Anthropic or Microsoft that they use it for ranking. It’s optional, not a win.
What we saw in our market check
In our November 16, 2026 check (36 AI answers across Columbus, Tampa and Boise), the answers cited 43 agent, team and brokerage websites. We fetched robots.txt for 40 of them, and none blocked OAI-SearchBot, GPTBot, ChatGPT-User, Googlebot, Google-Extended, Bingbot, ClaudeBot or PerplexityBot from the home page. That fits the obvious: sites that turn search crawlers away are much less likely to be cited at all. It doesn’t tell us how common blocking is in general; it only shows that the cited sites were open.
How can I check if it’s working?
- Google: in Search Console, open URL Inspection for your homepage and run the live test. It shows whether Googlebot can fetch and render the page. (Not set up yet? That’s win #8.)
- ChatGPT: wait about a day after any robots.txt change, then ask a question your site clearly answers (“[your name] realtor [town]”) and see whether your site is cited. Repeat it a few times; answers vary.
- Bing and Copilot: Bing Webmaster Tools’ AI Performance report shows when your pages are cited.
- Cloudflare sites: AI Crawl Control shows which AI crawlers actually visit (Cloudflare).
A quick tip: tools that “fake” a crawler (for example sending a request that claims to be OAI-SearchBot) are only a rough check, because CDNs verify real bots by their network addresses.
Want us to do it for you?
For members, we set robots.txt and firewall bot rules on purpose for each site, make sure the key facts are in the page text, and re-check after platform or hosting changes. We pick what fits each site, test it and adjust. Ask us and we’ll tell you exactly which crawlers can reach your site today.
Ask us to check your site · Next: Quick win #3: make your profiles match everywhere · All 10 quick wins
Sources
- OpenAI — Overview of OpenAI crawlers (accessed Nov 16, 2026) — OpenAI: OpenAI crawler overview
- OpenAI Help Center — Advertiser guidance for allowing OpenAI web crawlers (updated about Nov 6, 2026) — OpenAI: 20001243 advertiser guidance for allowing openai web crawlers
- Google — List of Google’s common crawlers (Google-Extended) — Google: google common crawlers
- Google Search Central — AI features and your website (updated Dec 10, 2025) — Google: ai features
- Google Search Central — Guide to optimizing for generative AI features (updated Jul 10, 2026) — Google: ai optimization guide
- Cloudflare — AI crawler and agent access policies (updated Nov 2, 2026) — Cloudflare: Cloudflare AI bot settings
- Cloudflare — AI Crawl Control (updated Aug 14, 2026) — Cloudflare: ai crawl control
- Anthropic — Does Anthropic crawl data from the web? — support.claude.com: 8896518 does anthropic crawl data from the web and how can site owners block the crawler
- Perplexity — Perplexity crawlers — docs.perplexity.ai/guides/bots
- Virtual Results — 36-answer market check with robots.txt check of 40 cited sites, November 16, 2026
