Why Your Website Is Invisible to ChatGPT (And How to Fix robots.txt)

Try this right now. Open ChatGPT in another tab and ask: "Tell me about [your business name]."

If the answer is vague, wrong, or confidently made up, you've just confirmed what most small business owners don't realise yet: the AI systems your customers are starting to use cannot see your website. And the fix is usually a single text file you've probably never looked at.

Why this matters more each quarter

AI search isn't a future problem — it's happening now. Google AI Overviews sit above the organic results for a growing share of queries. ChatGPT's web-search feature cites sources directly. Perplexity has built its entire product around answering with citations. When someone asks these systems "who are the best NDIS social workers in Sydney" or "plumbers in Brisbane who handle renovations", they get a curated answer that may or may not include your business.

If your website is blocked from AI crawlers, you're not in the running. You don't get cited. You don't get the click. Even when you rank well on traditional Google, you can be invisible on the AI layer sitting above it.

The one-file cause: robots.txt

robots.txt is a plain text file at the root of every website — yourdomain.com/robots.txt — that tells automated crawlers what they're allowed to access. When a crawler like GPTBot shows up, it reads this file first. If the file tells it to go away, it goes away. Simple as that.

The problem: many platforms ship with AI crawlers blocked by default, and many custom-built sites forget to include the crawlers at all — which also ends up excluding them on some configurations. Either way, the outcome is the same: AI systems can't read your content, so they can't cite your content.

How to check if your site is blocked

Paste your domain into this URL in any browser:

You're looking for any mention of AI crawlers. The specific bot names to scan for:

• GPTBot — OpenAI's training crawler
• ChatGPT-User — what ChatGPT uses when users ask it to read a page in real-time
• ClaudeBot — Anthropic's Claude
• PerplexityBot — Perplexity's search crawler
• Google-Extended — specifically controls AI Overviews + Gemini (separate from regular Googlebot)
• Applebot-Extended — Apple Intelligence + Siri

If you see any of these followed by Disallow: /, that crawler is blocked. If you see no robots.txt at all, the default behaviour is "allow everyone" — which is actually fine for AI crawlers, but it's still better to be explicit.

The fix — copy and paste this

Here's the full robots.txt that explicitly welcomes every major AI crawler while still allowing Google, Bing, and the rest of traditional search:

Replace yourdomain.com in the last line with your actual domain, then upload to the root of your site (same folder as your homepage).

Platform-specific guide

• Squarespace: you can't directly edit the file, but you can toggle AI crawler access under Settings → Crawlers & Spiders. Turn every toggle on.
• WordPress: plugins like Rank Math or Yoast let you edit robots.txt without touching files. Or edit directly via your hosting file manager.
• Wix: Settings → SEO Tools → Robots.txt. Paste the block above.
• Shopify: theme code → robots.txt.liquid. Merchants on newer plans can edit this directly.
• Custom build (static HTML, Vercel, Netlify, etc.): drop a file literally named robots.txt into your public directory. Done.

robots.txt is necessary — but not sufficient

Unblocking the crawlers gets them to your door. To actually get cited in AI answers, the content behind the door needs to be readable and valuable.

That means three more pieces working together:

• Valid schema markup: structured data (WebSite, Organization, FAQPage, Article, LocalBusiness) helps AI systems understand what your page is actually about. Schema score below 20/100 is common — and it's the difference between "indexed" and "actually cited".
• Clear, answerable content: AI systems love FAQ sections, clear H1/H2 hierarchies, and direct answers to specific questions. Vague "about us" pages don't get cited. Named experts, specific credentials, and concrete claims do.
• E-E-A-T signals: experience, expertise, authoritativeness, trust. Named authors with real credentials, proper contact info, reviews, and named clients all matter. For healthcare, finance, or legal content (YMYL topics), this is non-negotiable.

Real example: from 0 citations to AI-ready

When we audited Create Allied Health Services, a Sydney NDIS provider, their Squarespace site had every major AI crawler blocked by default. The founder is an AASW-registered clinical social worker and PhD candidate — exactly the kind of credentialed expertise AI systems want to cite — but no AI system could read her website.

Part of the rebuild we delivered included an explicit AI-friendly robots.txt, validated schema for LocalBusiness and MedicalBusiness, plus 48+ FAQ entries spread across service pages. That combination took the AI search readiness score from 41/100 to 85/100 — measurable because the pre-rebuild audit captured exactly where every score stood.

Similarly, our Functional Patterns Brisbane migration explicitly unblocked AI crawlers as part of the move off Squarespace. The site had zero ChatGPT citations before the cutover despite ranking top-10 on Google for high-volume biomechanics queries. Fixing the crawler access was a prerequisite for capturing any of that AI-search upside.

How to verify the fix worked

Once your updated robots.txt is live:

1. Visit yourdomain.com/robots.txt in an incognito window — confirm the new content loads.
2. Check Google Search Console's robots.txt tester to confirm it parses correctly.
3. Wait 2–4 weeks, then ask ChatGPT about your business again. If it cites you or at least gives accurate details, you're in the pipeline.

There's no "submit URL" button for AI crawlers the way there is for Google. GPTBot and the others re-crawl on their own schedules, usually every few weeks. Patience is part of the deal.

When "allowed" isn't enough

If you've checked the crawlers are welcomed, added schema, and still aren't surfacing in AI answers after a month, the issue is usually one of three things:

• Weak content signals: the page isn't actually useful for the queries you want to appear in. AI systems prefer concrete, specific content over generic marketing copy.
• Domain age + authority: newer sites and sites with few external links take longer to build citation trust. Not fixable overnight.
• Competing, better-optimised sites: your competitors have already done this work and AI systems are citing them instead. Time for a deeper AI search audit.

Next steps

If you've read this far, the two-minute version is:

1. Go to yourdomain.com/robots.txt right now.
2. If AI crawlers are missing or blocked, paste in the block above.
3. Check your schema status (most small business sites are under 20/100 — worth fixing).
4. Add FAQ content to your service pages if you don't already have it.

That's 90% of the AI-search visibility fight. The rest is content quality and time.