The data behind three studies of Melbourne business websites.
Three rounds between 17 and 18 September 2026 checked 278 businesses with a website — eight trades and clinic types, across six councils of outer Melbourne — for how fast the site shows on a phone, whether the phone number on it actually works, and whether AI assistants are allowed to read it. No business is named anywhere in what we publish. Every figure on this page, and every file below, is free to reuse with attribution under CC BY 4.0.
01 · First round
89 sites in Casey and Cardinia, timed on a phone.
The round that started it: business websites across Casey and Cardinia, loaded the way a customer's phone would and checked for a phone number a phone can actually dial. The pattern held clearly enough that we widened the sample to the rest of Melbourne's growth corridor.
02 · Full sample, all three rounds
Most of these sites are slow, and a quarter can't be tapped to call.
Read the full study: the Melbourne growth corridor, 278 sites →
03 · AI crawlers
59 of 273 sites refuse at least one AI assistant, and none says so in robots.txt.
By trade the refusal rate spreads only 14%–26%, which the underlying study treats as noise: the split runs by server software, not by trade. And requests come from an ordinary Australian connection, not from OpenAI's or Anthropic's published address ranges, so a refusal here means the site refuses anything calling itself that agent — an upper bound on what the verified crawler meets, not a certainty.
Downloads
Every figure, as a file.
18 files: the two combined rounds, the AI crawler round as CSV and as JSON, and every council and trade broken out on its own. Each is generated from the same underlying data as the study pages, so a number here always matches the number in the story.
| Dataset | Format | File |
|---|---|---|
| All three rounds combined (278 businesses) | CSV | /study/melbourne-growth-corridor-websites.csv |
| Round one: Casey and Cardinia (89 businesses) | CSV | /study/casey-cardinia-websites.csv |
| AI crawler refusals — summary by trade, council and server software | CSV | /study/websites-that-refuse-ai-crawlers.csv |
| AI crawler refusals — full detail (status codes, robots.txt cross-check) | JSON | /data/ai-crawler-refusals.json |
| Cardinia Shire business websites | CSV | /study/cardinia-business-websites.csv |
| City of Casey business websites | CSV | /study/casey-business-websites.csv |
| City of Hume business websites | CSV | /study/hume-business-websites.csv |
| City of Melton business websites | CSV | /study/melton-business-websites.csv |
| City of Whittlesea business websites | CSV | /study/whittlesea-business-websites.csv |
| City of Wyndham business websites | CSV | /study/wyndham-business-websites.csv |
| Melbourne chiropractors: their websites | CSV | /study/melbourne-chiropractor-websites.csv |
| Melbourne dentists: their websites | CSV | /study/melbourne-dentist-websites.csv |
| Melbourne electricians: their websites | CSV | /study/melbourne-electrician-websites.csv |
| Melbourne landscapers: their websites | CSV | /study/melbourne-landscaper-websites.csv |
| Melbourne physios: their websites | CSV | /study/melbourne-physiotherapist-websites.csv |
| Melbourne plumbers: their websites | CSV | /study/melbourne-plumber-websites.csv |
| Melbourne podiatrists: their websites | CSV | /study/melbourne-podiatrist-websites.csv |
| Melbourne roofers: their websites | CSV | /study/melbourne-roofer-websites.csv |
Method
How the sample was drawn. For each of eight trade and clinic categories in each of twelve suburbs across six councils (Cardinia Shire, City of Casey, City of Hume, City of Melton, City of Whittlesea and City of Wyndham), we searched the web for the category and suburb and took the first three businesses in the results with their own website, skipping directory listings, national chains and franchise networks. It is a sample of the businesses a customer searching would find first, not a census, and every group we publish holds at least two sites.
How speed was measured. Each home page was loaded with Google Lighthouse 12's mobile profile: a simulated mid-range phone on a throttled connection, one run per site. "Time to show" is Largest Contentful Paint (LCP), the moment the main part of the page appears. Google rates 2.5 seconds or less as good.
How the crawler round was run. Each home page was asked four times: once as a desktop browser, then once each as
GPTBot, OAI-SearchBot and ClaudeBot, using the user-agent strings OpenAI and Anthropic publish. A
site counts as refusing an agent when it answered the browser and did not answer that agent. Every refusal was then re-asked in the
opposite order — the assistants first, the browser last — at least five seconds apart, so a burst rate-limit is not recorded as a policy;
only refusals seen both times are counted. robots.txt was read in a separate survey of the same sites on 17 September, the day before, which
is how we know none of the refusers declares a block there.
The upper-bound caveat. Our requests come from an ordinary Australian connection, not from OpenAI's or Anthropic's published address ranges. A refusal in this data means the site refuses anything calling itself that crawler; a filter that checks the address as well might let the real one through. That makes the refusal counts an upper bound on what the verified crawler meets, and the honest number somewhere at or below it.
Citing this data
Everything on this page and every file linked from it is published under CC BY 4.0: quote it, chart it, republish it, for any purpose, including commercial — credit Jaguars Digital and link back to this page.
Corrections
If a figure here is wrong, tell us and we correct it in public, with the date it changed. 0482 022 493, where you speak to a real person.