← Blog robots.txt GPTBot Data Technical
Accountants block GPTBot three times more often than plumbers
2026-07-21 · Martin Nymann · 5 min reading
A single line of code in robots.txt can make you invisible in ChatGPT. We measured how widespread this is across 465 Danish company websites — and how you can check your own in 10 seconds.
A single line in robots.txt can make your website invisible in ChatGPT. Of 465 Danish business websites scanned on 21 July 2026, 23 sites (5 per cent) blocked GPTBot. Accountants do so three times more often than plumbers: 8.8 per cent versus 3.5 per cent.
How many block AI crawlers?
| Crawler | Accountants | Plumbers | Solicitors |
|---|---|---|---|
| GPTBot (ChatGPT) | 13 (8.8%) | 7 (3.5%) | 3 (2.5%) |
| Google-Extended (Gemini) | 11 (7.5%) | 7 (3.5%) | 2 (1.7 per cent) |
| ClaudeBot | 7 (4.8%) | 8 (4.0%) | 2 (1.7%) |
| PerplexityBot | 4 (2.7%) | 5 (2.5 per cent) | 2 (1.7%) |
Is this a deliberate choice?
Our data cannot determine the intention — only that the block is in place. But the pattern is worth noting: auditors most often block both GPTBot and Google-Extended, i.e. the two crawlers most commonly found in ready-made robots.txt templates and in ‘block AI’ plugins. This points to templates and plugins rather than a deliberate decision.
If the block is deliberate — for example, because you do not want your material included in training data — then it is correctly set. If it has been inherited from a template, it will cost you mentions in AI responses without anyone having chosen it.
How to check your own robots.txt
- Open
dittfirma.dk/robots.txtin your browser. - Search for
GPTBot,ClaudeBot,PerplexityBotandGoogle-Extended. - If it says
Disallow: /under one of them, that particular crawler is blocked from the entire site.
Note the difference: Disallow: /minmappe/ only blocks a folder and is rarely a problem. It is Disallow: / that closes the whole door.
This article is based on the same dataset as our main analysis of 465 Danish corporate websites, where you’ll find the full methodology and all the figures.
Method
Data was collected by Geoa on 21 July 2026. We fetched the homepage of 465 Danish company websites without running JavaScript — exactly as GPTBot, ClaudeBot and PerplexityBot do — and measured six technical signals: readable text, llms.txt, robots.txt access for AI crawlers, structured data (schema.org), H1 heading and meta description. The breakdown is 200 plumbing firms, 147 accountants/bookkeepers and 118 solicitors.
Important caveat: the sample consists of search-visible businesses — those that rank in local searches — not a random cross-section of the industry. The figures therefore describe the industry’s most visible businesses. A representative sample would, in all likelihood, look worse, not better. We therefore consistently write ‘of those scanned’, not ‘of all Danish’.
How does your own website fare?
Run the free AI Readability Check and see exactly what AI crawlers are reading on your website — and where you stand within your industry. No login required.
Check my site for free →Get your free GEO Score — 60 seconds, no credit card required.
Get started for free