133 of 10,099 Shopify stores block an AI crawler. 6 block the one ChatGPT shops with.

We read the robots.txt of 10,099 Shopify storefronts for twelve AI crawlers. 1.32% block at least one there, and that is a floor: a block at the edge is invisible to this method. Nearly all of it is aimed at training crawlers, by one whole-site rule, and mostly by one copied list. The four answer-time crawlers we checked, OpenAI’s and Perplexity’s, are blocked by 13 stores.

The question

Merchants are told two opposite things about AI crawlers: block them, because they take your content and give nothing back; and admit them, because that is how a store gets into an AI shopping answer. Both pieces of advice assume a decision is being made. We wanted to know how many stores have made one, and which way.

Method

Every store in a corpus of 10,099 known Shopify storefronts has its /robots.txt read as part of a scan. The file is parsed the way the major crawlers document parsing it: the most specific User-agent group wins and * is the fallback; within a group the longest matching path rule wins and Allow beats Disallow on a tie. Each of twelve crawler names is then asked one question: may it fetch /products/? A store counts as blocking a crawler when the answer is no.

A second reading, separate from robots.txt, looks at the product page for a meta name="robots" tag carrying noai or noimageai.

Readings were taken between 29 August 2026 and 2 September 2026; the corpus count is as of 2 September 2026. Every store that blocked at least one crawler, or carried the tag, is one row in the data linked below. The rest of the corpus blocked nothing and carried nothing, and is the denominator.

What this method cannot see. A robots.txt is a request. A store can also block a crawler at the edge, with a bot-management rule or a firewall, and that block is invisible here because the scanner is not the crawler being blocked. So every count on this page is a lower bound on blocking. A store that blocks by robots.txt is certainly blocking; a store that does not may still be.

Result

How they block

• 130 of 133 block with Disallow: / — the whole site, not the product pages. 3 block /products/ alone.
• 54 block exactly one crawler.
• 28 block the same eight, verbatim: Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, GPTBot, Google-Extended, meta-externalagent. 4 more block those eight and others. Identical lists do not arise independently; this one reads like a snippet that was copied, and it contains none of the four answer-time crawlers.
• 1 store blocks all twelve.

Training or answering

Of the 133 stores that block anything, 120 block only training and grounding crawlers. 13 block at least one crawler that fetches at answer time; 1 of those blocks only answer-time crawlers and 12 block both kinds.

The clearest pair is OpenAI’s. GPTBot trains; OAI-SearchBot is what ChatGPT’s search and shopping results are built from. 77 stores block the first. 6 block the second, and every one of those 6 also blocks the first. The reverse — keep the shopping crawler out, let the training crawler in — happens on 0 stores. No store in the corpus has decided to stay out of AI shopping answers. The stores that are out of them by robots.txt are there because a copied training opt-out happened to include the name.

The noai tag

8 stores carry a noai or noimageai meta robots tag on their product page. 6 of the 8 block nothing at all in robots.txt. By their names, six of the eight are musicians’ merchandise stores, which again looks like one template rather than eight decisions. The tag is a separate mechanism from robots.txt and this study makes no claim about which crawlers honour it.

What this does and does not mean

Blocking AI crawlers is rare on Shopify. 98.7% of stores block none of the twelve by robots.txt. Where it happens it is mostly a whole-site rule aimed at training crawlers, and mostly one list.

The shopping crawlers are almost never blocked. 13 stores in 10,099 block any of the four answer-time crawlers we checked. For the rest, whatever stands between the store and an AI shopping answer, it is not robots.txt.

It is not a recommendation. A merchant who does not want their catalogue in a training set has a reason, and a copied list that does the job is a fine way to act on it. The finding is narrower: the list most of them copied was written before the answer-time crawlers existed, so a store using it has made a decision about training and, without noticing, none about shopping.

It is a lower bound. See the method. Edge blocks are real and this cannot count them.

Reproduce it

One store, one file, no tooling. Pick any Shopify domain:

curl -s https://EXAMPLE.com/robots.txt

Find the User-agent: group that names the crawler you care about; if none does, the * group applies. Under that group, the longest rule matching /products/ decides, and Disallow: / matches everything.

All 139 rows, one per store: ai-crawler-blocking.csv — host, grade, date read, one column per crawler (allowed or blocked), whether the noai tag was present, and whether the blocking rules were whole-site or product pages only. Every row names a domain whose robots.txt you can read yourself.

Reuse the figure

CC BY 4.0. The image is served from here and carries its own source line; the snippet wraps it in a link back to this page.

Who ran this

Shelfglance, which measures what AI assistants can read from Shopify storefronts and keeps a public directory of 10,099 of them. The per-crawler check on every store report is the same reading this page adds up. Other studies are on the findings page and in the feed.