Right now, many Shopify Plus store owners are unknowingly blocking the next generation of buyer traffic by using outdated security configurations that flag AI retrieval bots as malicious scrapers. The AI visibility platform Pendium helps brands identify these hidden access blocks, which typically occur when Cloudflare Web Application Firewall rules or default Shopify robots.txt setups catch agents like GPTBot and ClaudeBot in their automated security nets. To solve this problem, merchants must implement precise WAF rule overrides in Cloudflare and deploy a custom robots.txt.liquid template in Shopify to explicitly allow verified retrieval search bots while keeping bad actors locked out.
The difference between training scrapers and retrieval agents
AI bots are not a monolith. They fall into two distinct camps with opposing intents, and treating them all as hostile threats is a major unforced error for modern ecommerce brands.
Training scrapers (like Bytespider or CCBot) crawl the web to build massive datasets for future model training. They consume server bandwidth and copy your product descriptions, blog posts, and reviews without directing any immediate traffic back to your store.
Retrieval agents (such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot) act as search engines. They crawl your site to answer real-time shopping queries from users on platforms like ChatGPT, Claude, and Perplexity. If an enterprise buyer asks an AI agent for a specific product recommendation, these bots must be able to read your live storefront to include you in the recommendation.
| Bot Category | Example User-Agents | Core Function | Action for Ecommerce |
|---|---|---|---|
| Retrieval Agents | OAI-SearchBot, PerplexityBot, Claude-SearchBot | Resolves live user product queries and generates citations. | Allow |
| Training Scrapers | GPTBot, ClaudeBot, Google-Extended, Bytespider, CCBot | Gathers dataset content for future LLM training. | Conditional (Block or Throttle) |
Blocking retrieval bots means your store does not exist in the datasets these engines use to answer high-intent buying questions. For merchants focused on customer acquisition, this visibility gap is a silent leak of highly qualified buyer traffic.
Auditing your Cloudflare bot management settings on Shopify Plus
Most high-volume Shopify Plus stores route their traffic through a Cloudflare proxy for web security. While these defenses are necessary to prevent malicious inventory scraping and checkout abuse, default settings often block legitimate retrieval bots.
Checking the "Block AI bots" toggle
Cloudflare offers a one-click setting on its Security Settings page under Bot Traffic labeled "Block AI bots." While convenient, this toggle is a blunt instrument that blocks training crawlers and retrieval agents alike.
If "Block AI bots" = Enabled
Then GPTBot, OAI-SearchBot, and ClaudeBot -> 403 Forbidden
When this toggle is active, any search agent attempting to index your catalog receives a 403 challenge page instead of your product HTML. This instantly breaks your store's crawlability, removing your products from real-time AI search results.
Understanding bot scores and challenges
Under the hood, Cloudflare's Bot Management engine scores every incoming request from 1 to 99 to assess its likelihood of being automated, as detailed in the Cloudflare WAF docs. A score of 1 indicates a definitely automated script, while high scores represent human browsers.
New or unverified retrieval agents often trigger low bot scores because they run on distributed, non-standard cloud infrastructure. If your WAF is configured to challenge or block all low-scoring automated traffic, it will stop headless AI crawlers cold. Because these crawlers cannot solve an hCaptcha, they abandon the crawl and mark your catalog as offline.
Writing the WAF custom rule exceptions
To maintain strong security without sacrificing your AI discoverability, you must write a custom WAF rule in Cloudflare. This rule bypasses bot-blocking rules specifically for verified search and retrieval agents.
Before opening your store to these crawlers, you should verify that once they get in, they can read clean, accurate pricing data. Ensuring your schema is correct prevents AI search engines from indexing outdated or incorrect numbers. You can learn more about formatting this data in our guide to Fix Shopify tax-inclusive JSON-LD so AI displays actual prices.
To configure the WAF exception, log into your Cloudflare dashboard and navigate to Security -> WAF -> Custom Rules. Create a rule with the following parameters:
- Rule Name: Allow Verified AI Search Bots
- Field:
http.user_agent - Operator:
matches regex - Value:
(OAI-SearchBot|Claude-SearchBot|PerplexityBot|Perplexity-User|ChatGPT-User)
Configure the action for this rule to Bypass and select Bot Management. This ensures that Cloudflare skips the bot score evaluation for these specific user-agents, letting them reach your Shopify storefront without triggering CAPTCHAs.
Rule Expression:
(http.user_agent matches "(OAI-SearchBot|Claude-SearchBot|PerplexityBot|Perplexity-User|ChatGPT-User)")
Action: Bypass (Bot Management)
This targeted rule keeps your defensive shield up against hostile scrapers and botnets while allowing legitimate buying engines to read your collections and product specs.
Overriding Shopify's default robots.txt generation
Even if Cloudflare lets the bots pass, Shopify's native robots.txt file can still lock them out of your product pages. By default, Shopify auto-generates a robots.txt that applies broad rules under the wildcard User-agent: *. This configuration blocks standard system paths like /cart, /checkout, and administrative endpoints, which is correct for security.
However, the default file lacks explicit instructions for newer AI user-agents. Some third-party applications or legacy templates inject aggressive wildcard disallows to manage duplicate content, which can accidentally block AI agents from crawling product pages.
Creating the robots.txt.liquid template
To take control of how bots crawl your site, you must override the default file. In Shopify Plus, you can customize this behavior by editing the theme code to generate a custom template, as outlined in Shopify Robots.txt for AI Crawlers.
To create the template:
- From your Shopify admin, go to Online Store -> Themes.
- Click the three dots next to your active theme and select Edit code.
- Under Templates, click Add a new template.
- Select robots from the dropdown and click Create template.
Shopify will generate a templates/robots.txt.liquid file prepended with a default Liquid loop. Keep this loop intact, as it preserves Shopify's automatic system blocks for sensitive paths.
The exact allow rules to deploy
To explicitly allow retrieval agents to scan your catalog, you must append custom rules directly below the default loop. While editing this template, it is also a good opportunity to structure your product groups so bots can parse your collections cleanly. For step-by-step guidance on structuring your categories, refer to our guide on how to Map your Shopify product taxonomy for AI search agent recommendations.
Add the following code to the bottom of your robots.txt.liquid file:
# Default Shopify rules loop
{%- for group in robots.default_groups -%}
{{- group.user_agent }}
{%- for rule in group.rules -%}
{{ rule }}
{%- endfor -%}
{%- if group.sitemap != blank -%}
{{ group.sitemap }}
{%- endif -%}
{%- endfor -%}
# Custom overrides for AI Search & Retrieval Bots
User-agent: OAI-SearchBot
Allow: /products/
Allow: /collections/
Allow: /blogs/
Disallow: /cart
Disallow: /checkout
Disallow: /admin
User-agent: Claude-SearchBot
Allow: /products/
Allow: /collections/
Allow: /blogs/
Disallow: /cart
Disallow: /checkout
Disallow: /admin
User-agent: PerplexityBot
Allow: /products/
Allow: /collections/
Allow: /blogs/
Disallow: /cart
Disallow: /checkout
Disallow: /admin
Save the template. The changes will deploy instantly to your root directory. You can verify the update by visiting yourstore.com/robots.txt in your browser to confirm the rules are live.
Verifying your configuration with curl
Once your Cloudflare and Shopify rules are deployed, you must verify that the bots can successfully access your pages. The most reliable way to test this is by simulating bot requests using terminal curl commands.
Run this command in your terminal, replacing yourstore.com with your actual domain and /products/test-product with a live product path:
curl -A "OAI-SearchBot" -I https://yourstore.com/products/test-product
Review the response header returned by your server. You want to see an HTTP status code of 200 OK:
HTTP/2 200
content-type: text/html; charset=utf-8
server: cloudflare
If the command returns a 403 Forbidden or redirects to a Cloudflare challenge page, your custom WAF rule is not configured correctly. If it returns a 404 Not Found or matches your home page, double-check that Shopify is not redirecting the user-agent.
Testing with multiple user-agent strings—such as Claude-SearchBot and PerplexityBot—ensures your site is fully accessible to the main engines driving conversational commerce traffic.
To find out if ChatGPT, Claude, and Gemini are currently blocked or able to recommend your catalog, you can run a quick check using the AI Site Audit — Is Your Website Ready for AI Agents? tool. For a comprehensive look at your brand's standing, run your URL through Pendium's free AI Visibility Scan at Pendium.ai to see exactly how your brand is perceived across all 7 major AI platforms in under 2 minutes.