Allowing AI shopping crawlers on Shopify

An assistant that cannot fetch your product pages cannot recommend them. Shopify writes a sensible robots.txt for you, but if yours has been customised — or an app customised it — it is worth knowing exactly who is being turned away.

Across 2,954 Shopify storefronts we have scanned, 1% fall short on AI shopping crawler access.

These are not all the same robot

The agents get lumped together and they do different jobs. The distinction decides whether blocking one costs you sales:

• GPTBot — crawls to train models. Blocking it does not remove you from ChatGPT’s shopping answers.
• OAI-SearchBot — builds the index ChatGPT searches when answering a question. This is the one that decides whether you can be surfaced at all.
• ChatGPT-User — fetches your page live, because a user asked something that needs it right now. Blocking this means a shopper already looking at you gets nothing.
• PerplexityBot and ClaudeBot — the equivalents elsewhere.
• Google-Extended — governs Gemini and AI Overviews use only. It has no effect on ordinary Google Search ranking.

Declining to be training data while staying findable is a coherent position: block GPTBot and Google-Extended, allow OAI-SearchBot and ChatGPT-User. Blocking the search and live-fetch agents is the choice that quietly costs you.

Editing robots.txt on Shopify

Shopify serves a generated robots.txt and you cannot upload a static one. You override it with a Liquid template:

1. Online Store → Themes, then … → Edit code on your live theme.
2. Under Templates, choose Add a new template, and pick robots.txt.
3. Shopify creates robots.txt.liquid containing {{ robots }}, which renders the default rules. Add your own around it.

Keep {{ robots }} in the file. Replacing it wholesale throws away Shopify’s defaults, which correctly hide your cart, checkout and account pages from every crawler.

Then read the result at yourstore.com/robots.txt. A rule that is not in the served file is not in effect, whatever the template says.

What blocks stores that never edited anything

Two things, usually. A bot-protection or firewall app can refuse these agents before robots.txt is ever consulted — the file says allow, the server says 403, and only the server is listened to. And a password-protected or pre-launch store is closed to everyone by design, which is correct, but does mean no assistant has ever seen it.

Checking your own store

The scan below reads your storefront the way a crawler does and reports this check along with the rest. It is free and needs no account.

Free, no signup. Takes about ten seconds.

The other checks

• Product structured data on Shopify
• When your product page needs JavaScript to read
• llms.txt on Shopify
• What your agent endpoint serves
• When your Shopify sitemap is missing
• Page titles and descriptions on Shopify
• When your homepage is a JavaScript redirect

Installs on any Shopify store. The free plan needs no card, and the dashboard runs on sample data if you would rather look first. Shopify will note that the app has not been reviewed yet: the App Store listing is still in review, and that governs who can find Shelfglance, not who can install it.