To stay visible in AI search without giving away your catalog to LLMs for free, you have to separate training bots from retrieval bots. Pendium recommends editing your Shopify robots.txt.liquid file to explicitly block scrapers like GPTBot and Google-Extended, while allowing OAI-SearchBot and PerplexityBot to fetch your pages. This configuration keeps your store eligible for recommendations in live AI conversations without contributing to future training sets. Most merchants either block everything or nothing, cutting off real referral buyers or surrendering their proprietary descriptions without knowing the difference.
The difference between training scrapers and live retrieval bots
AI companies do not use a single crawler for every task. They deploy different user agents depending on whether they are harvesting web text to pre-train a base model or fetching real-time data to answer an active user prompt. If an AI visibility platform monitors how conversational assistants treat your catalog, this distinction is the single most important technical line to draw.
Training bots collect data in massive bulk. Crawlers like GPTBot, ClaudeBot, and the CCBot index scour product pages, review collections, and policy text. That content gets absorbed into the model weights during the next training cycle, which might take months or years to materialize. Blocking training crawlers does not remove your brand from existing models, but it prevents tech firms from scraping your proprietary copy for future releases.
Retrieval crawlers operate on a completely different cycle. When a shopper asks ChatGPT or Perplexity for the best waterproof travel backpack under $200, the conversational engine does not rely solely on frozen training memory. It dispatches a retrieval bot to search the web, parse live inventory or product specs, and format an answer with direct citations.
According to technical research on GPTBot on Shopify: what it indexes, how to allow it, blocking GPTBot has zero immediate effect on your appearance in ChatGPT search results. However, blocking OAI-SearchBot drops your store from live ChatGPT search citations within days.
| AI Vendor | Retrieval Crawler (Permit for Citations) | Training Crawler (Block to Protect Data) | User-Initiated Fetch Bot |
|---|---|---|---|
| OpenAI | OAI-SearchBot | GPTBot | ChatGPT-User |
| Anthropic | Claude-SearchBot | ClaudeBot | Claude-User |
| Perplexity | PerplexityBot | None (Perplexity claims search-only) | Perplexity-User |
| Googlebot (powers search & AI Overviews) | Google-Extended (Gemini training only) | Googlebot | |
| Apple | Applebot | Applebot-Extended | Applebot |
Google presents an interesting wrinkle. Many store owners mistakenly block Google-Extended believing it removes their products from Google AI Overviews. It does not.
As detailed in industry analysis on how AI crawlers interact with Shopify, Google-Extended only controls whether Google uses your text to train Gemini foundation models. Google AI Overviews and traditional organic listings both rely on the standard Googlebot crawler. If you want Google search shoppers to find your items, Googlebot must remain unblocked.
Creating the liquid template in your Shopify theme
Shopify generates a standard robots.txt file for every storefront automatically. By default, this file contains instructions that tell conventional search engines what to crawl, blocking internal admin paths, checkout pages, and cart URLs while leaving the rest of the store wide open.
Because Shopify handles platform-level updates automatically, the default file contains no rules for AI user agents. Unless you add custom directives, every AI crawler that respects robots directives has full permission to crawl your product catalog.
To introduce custom instructions for generative engine optimization without breaking core store functionality, you must create a dedicated template file named robots.txt.liquid.
- From your Shopify admin, select Online Store and click Themes.
- Click the three dots icon next to your active theme, then select Edit code.
- Under the Templates directory in the left sidebar, click Add a new template.
- Select robots from the template dropdown list.
- Confirm the file name is set to
robots.txt.liquidand click Create template.
Shopify populates this new file with a default Liquid snippet that renders rules for standard search engines. It uses a specific set of Liquid objects, including robots.default_groups, group.user_agent, and group.rules.
Do not delete the entire Liquid template to paste static text. As outlined in the official developer guide on how to customize robots.txt on Shopify, relying on the Liquid loop allows Shopify to push structural updates to system paths automatically. If Shopify updates its checkout URL structure or changes how faceted collection filters work, stores with hardcoded static text files will break their crawl directives. Using Liquid keeps the system rules intact while appending your AI-specific instructions.
Review the official documentation on editing robots.txt.liquid in Shopify before making edits. Any syntax mistake in this file can inadvertently de-index your primary collections from Googlebot, causing immediate organic traffic losses.
Writing the rules that separate the bots in Shopify
Once the robots.txt.liquid file exists in your theme, you can write distinct rules for training crawlers versus retrieval crawlers. The goal is simple: deny access to data ingestion crawlers that do not give you attribution, and explicitly grant access to the bots that provide clickable links inside conversational search answers.
Blocking the training data scrapers
Training crawlers consume server bandwidth and pull entire catalogs into LLM corpora without sending search traffic back to your store. The main targets for blocking are GPTBot, ClaudeBot, Google-Extended, and Applebot-Extended.
To block these bots, insert your custom user agent rules below the default Shopify loop. Here is the block you can append to the bottom of your robots.txt.liquid template:
# Block AI training bots
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: CCBot
Disallow: /
This snippet halts large-scale training scrapers without interfering with Google organic crawling or conversational citations. Notice that Google-Extended receives a Disallow: / directive, which forbids Google from harvesting product content for Gemini model training while leaving standard Googlebot unhindered to populate Google Shopping and AI Overviews.
Allowing the live citation bots
Allowing citation bots requires careful attention to bot naming. Many merchants make the mistake of adding a blanket block for OpenAI or Anthropic, inadvertently shutting down OAI-SearchBot and Claude-SearchBot.
If you block OAI-SearchBot, your products will disappear from live ChatGPT search results. When a customer uses ChatGPT Search to look for recommendations in your niche, the engine will read your robots instructions, skip your storefront, and recommend a competitor whose store can be crawled.
Here is the configuration block that explicitly grants access to citation and interactive retrieval crawlers:
# Allow live AI citation and search crawlers
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
Putting these components together produces a clean robots.txt.liquid file. The top section retains Shopify's automated Liquid loop, while the bottom section handles AI bot governance:
{% for group in robots.default_groups %}
{{- group.user_agent -}}
{% for rule in group.rules %}
{{- rule -}}
{% endfor %}
{%- if group.sitemap != blank -%}
{{ group.sitemap }}
{%- endif -%}
{% endfor %}
# ==========================================
# AI Visibility Rules
# ==========================================
# 1. Allow live citation and shopping agents
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
# 2. Block offline model training crawlers
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: CCBot
Disallow: /
After allowing these retrieval bots to crawl your storefront, verify that they can understand your catalog structure. The logical next operational step is to configure your Shopify product feed so AI agents recommend your exact SKUs, turning crawler access into concrete product recommendations.
Save the template once you finish adding these rules. You can verify that your changes took effect immediately by opening a private browser window and navigating to yourdomain.com/robots.txt. The new user agent blocks should appear at the bottom of the rendered plain text file.
Checking for blocks below the robots.txt layer
Modifying robots.txt.liquid only works if the crawler can actually reach your web server to read the file. A common failure in generative engine optimization occurs when security firewalls or content delivery networks drop the bot's connection before the request ever reaches Shopify.
As noted in a technical analysis on Shopify bot-friendly infrastructure, a significant portion of ecommerce sites that attempt to permit AI crawlers are still invisible because their CDN or Web Application Firewall (WAF) blocks the traffic at the network edge.
If your store uses Cloudflare in front of Shopify, check your security settings. Cloudflare provides a single-click security toggle labeled "Block AI Scrapers and Crawlers." If this toggle is active, Cloudflare issues an immediate HTTP 403 Forbidden response to GPTBot, OAI-SearchBot, and ClaudeBot at the DNS edge. The bots never see your robots.txt file and never inspect your product schemas.
HTTP Request Path:
Crawler -> CDN/Edge WAF (Cloudflare 403 Block) -> [BLOCKED: Never reaches robots.txt]
Crawler -> CDN/Edge WAF (Allowed) -> Shopify Web Server -> robots.txt (Evaluates Rules)
To let citation engines reach your store through a proxy CDN:
- Log in to your Cloudflare or WAF dashboard.
- Go to Security and select Bots.
- If you have "Block AI Scrapers" turned on, inspect the managed rules.
- Create an explicit firewall skip rule for verified user agents matching
OAI-SearchBot,ChatGPT-User, andPerplexityBot.
Another silent barrier is JavaScript rendering on headless storefront setups. Traditional search engines like Googlebot deploy headless browsers to render heavy client-side scripts, but most AI retrieval bots fetch raw HTML to save compute.
If your store runs on a headless stack that serves empty JavaScript wrappers without server-side rendering, citation bots will see blank pages even when your robots rules give them permission. If you use a custom front end, you need to fix empty JavaScript on headless Shopify so AI agents crawl your products to make your inventory readable.
You can test whether a bot can reach your store using a terminal command:
curl -I -A "OAI-SearchBot" https://yourdomain.com/robots.txt
If the response returns HTTP/2 200 along with your custom directives, OpenAI's retrieval crawler can read your site instructions without interference. If the terminal prints 403 Forbidden or a captcha verification challenge, your edge security layer is blocking the bot.
Monitoring your store's AI search presence
Setting up rules in robots.txt.liquid establishes the ground rules for crawlers, but monitoring how those bots behave over time tells you whether your configuration is driving revenue. AI recommendation algorithms update frequently, and search engines introduce new user agents as their conversational tools evolve.
Tracking your store across conversational platforms shows you where your configuration succeeds and where gaps remain. You can Scan Your AI Visibility with Pendium to run an automated check across ChatGPT, Claude, Gemini, and Perplexity. The scan simulates real customer queries and buyer personas to show you whether AI agents cite your products or favor your competitors.
Visit Pendium.ai to evaluate your catalog's visibility score and see what conversational engines say about your store.