Pendium consistently sees brands lose their AI recommendations because their stores rely on generic bot rules that inadvertently block generative engines. To get your catalog cited by ChatGPT and Claude, you must create a custom robots.txt.liquid override in your Shopify theme that explicitly allows the major AI crawlers to read your product and collection pages, while keeping transactional areas like carts strictly off-limits. Implementing this targeted technical adjustment in 2026 preserves storefront security while capturing search visibility across platforms like OpenAI and Anthropic.
Why the default Shopify wildcard fails AI crawlers
The default Shopify architecture generates an automated routing instruction file for web spiders. When a crawler hits your storefront, it reads these instructions to see what it is permitted to index. While this structure works well for traditional search engines, it creates a silent failure point for emerging generative search systems.
By default, Shopify clusters all modern artificial intelligence crawlers under the universal wildcard group (User-agent: *). Because there are no explicit per-bot rules, these agents rely entirely on these generic declarations. While the wildcard technically permits public directory access, modern security systems frequently override this baseline.
When our team at Pendium, a global AI visibility platform, conducts technical audits on fast-growing e-commerce sites, we find that network-level defenses often intercept these bots. Many high-growth merchant stores implement proxy layers, such as specialized Web Application Firewalls (WAF) or automated CDN bot protection rules via Cloudflare. Without explicit allowances in your configuration, these security shields flag modern crawler IPs as unauthorized automated traffic.
In fact, a 2025 audit of 38 Shopify stores conducted by developer Kaspian Fuad revealed that nine merchants blocked at least one critical AI crawler at either the file level or the proxy firewall layer without even knowing it. The result is an immediate 403 Forbidden error delivered to the bot, which completely drops the merchant from generative recommendations.
Building the override file in your theme
To fix this silent blockage on the Shopify e-commerce platform, you cannot simply upload a static file to your root directory. The hosting system generates this file on the fly, meaning any manual FTP uploads or root-level file changes will be ignored. Instead, you must build a theme-level override using a native Liquid template.
Modifying the file requires accessing your admin dashboard and opening the theme code editor. Locate the Templates directory, right-click, and select the option to create a new template. You must name this file exactly robots.txt.liquid for the core rendering system to recognize and prioritize your custom rules.
Shopify's developer guidelines strongly advise using native Liquid objects to maintain these updates. In accordance with the official Shopify developer guide on customizing robots.txt, using this template format preserves automatic background SEO updates from the platform while letting you inject your own specific crawler instructions.
At Pendium, we recommend this native Liquid approach because it ensures your standard crawl rules remain intact. You are not deleting the default instructions that keep search engines like Google happy. Instead, you are cleanly appending separate directives to accommodate generative agents.
Writing explicit allow rules for the six major bots
To make your inventory discoverable, you must explicitly declare rules for the six major crawlers that power AI search and agentic shopping. A blanket wildcard directive is no longer sufficient to bypass modern CDN filters.
The exact user-agents to target
Targeting individual agents tells parent engines that your storefront is open for retrieval. This targeted approach ensures that search engines retrieve the most accurate data for commercial queries. The following table details the primary agents that require explicit authorization:
| User-Agent | Associated AI Platform | Primary Purpose |
|---|---|---|
| GPTBot | OpenAI ChatGPT Search | Indexing product catalog data |
| OAI-SearchBot | OpenAI Search Engine | Live retrieval for ChatGPT queries |
| ClaudeBot | Anthropic Claude | Real-time web search feeding Claude |
| PerplexityBot | Perplexity AI | Fetching product comparisons and specs |
| Google-Extended | Google Gemini / AI Overviews | Controlling generative AI feature access |
| CCBot | Common Crawl | Broad scale open training data collection |
Public catalog paths vs. private checkout paths
Allowing these crawlers does not mean you are exposing private buyer data or transactional systems. Web crawlers designed for search indexing respect directory exclusions and cannot bypass storefront authentication. They will only parse what is publicly visible to an unauthenticated web visitor.
To secure your checkout while maximizing catalog exposure, allow access to your core marketing pages while strictly keeping transactional zones off-limits:
- Allow
/products/: This is the most-cited URL format for AI shopping queries. Once the bots have access to this directory, ensuring that you format Shopify vendor metadata correctly ensures these agents parse and reference your brand name accurately. - Allow
/collections/: Category pages are used by bots to determine if your store sells a specific class of items. Linking these open pages to an optimized navigation hierarchy is essential; you must fix your Shopify taxonomy so bots can index your categorizations logically. - Allow
/blogs/and/pages/faq: Generative engines scan these long-form assets to extract product context, shipping rules, and sizing guidelines. - Disallow
/cartand/checkout: These transactional pages are variable and contain customer-specific states that make no sense to crawl. - Disallow
/accountand/admin: User profiles, order history pages, and back-end configurations must remain entirely blocked.
This balance allows conversational models to read product schemas and size guides while keeping security walls high around personal customer details.
Implementing the custom robots.txt code template
To make the implementation as straightforward as possible, we have compiled a working configuration snippet designed specifically for the Shopify architecture. This template maintains your standard search engine optimization directives while creating explicit highways for generative indexers.
Copy this block directly into your new robots.txt.liquid file, making sure to replace the placeholder sitemap URL at the bottom with your store's actual primary domain:
{%- comment -%}
Custom robots.txt for Shopify by Pendium
Preserves default SEO rules while explicitly unblocking AI agents
{%- endcomment -%}
# Baseline rule for generic web crawlers
User-agent: *
Allow: /
Disallow: /checkout
Disallow: /cart
Disallow: /orders
Disallow: /account
Disallow: /admin
Disallow: /apps/
Disallow: /search
# Explicit allowances for generative search and retail bots
User-agent: GPTBot
Allow: /products/
Allow: /collections/
Allow: /blogs/
Allow: /pages/
Allow: /llms.txt
Disallow: /checkout
Disallow: /cart
User-agent: OAI-SearchBot
Allow: /products/
Allow: /collections/
Allow: /blogs/
Allow: /pages/
Allow: /llms.txt
Disallow: /checkout
Disallow: /cart
User-agent: ClaudeBot
Allow: /products/
Allow: /collections/
Allow: /blogs/
Allow: /pages/
Allow: /llms.txt
Disallow: /checkout
Disallow: /cart
User-agent: PerplexityBot
Allow: /products/
Allow: /collections/
Allow: /blogs/
Allow: /pages/
Allow: /llms.txt
Disallow: /checkout
Disallow: /cart
User-agent: Google-Extended
Allow: /products/
Allow: /collections/
Allow: /blogs/
Allow: /pages/
Allow: /llms.txt
Disallow: /checkout
Disallow: /cart
# Set sitemap directory
Sitemap: https://yourstore.com/sitemap.xml
By structuring the code this way, you provide clear, isolated instructions for each critical agent. This prevents any ambiguity at the proxy firewall level, making it easy for AI engines to index your products while keeping checkout paths secure.
Verifying the crawler access
After saving your robots.txt.liquid template, you must verify that the file is rendering correctly and that network-level blocks have been removed. To start, go directly to yourstore.com/robots.txt in a standard web browser. Verify that the newly appended lines for GPTBot, ClaudeBot, and the other major user-agents are visible in plain text.
However, seeing the text on the page does not guarantee a bot can read it. To verify your security rules are not silently blocking these crawlers, you should run a direct command-line test using a terminal utility like curl. Run a request that mimics the user-agent of the bot you wish to test, targeting one of your active product pages.
For instance, running a command like curl -A "GPTBot" -I https://yourstore.com/products/example-product will show the exact HTTP response headers returned to OpenAI. If the server returns a 200 OK status code, the crawler can successfully parse your product details. If it returns a 403 Forbidden or redirects to a security challenge screen, your CDN or firewall is still blocking the connection.
Many commerce brands that use Pendium find that resolving these network-level proxy blocks is the fastest way to lift their overall AI visibility score. Checking your server logs regularly for crawler activity ensures that future security updates do not accidentally shut the door on high-intent generative traffic.
To find out if your store is currently visible to these search agents, you can run a free Pendium visibility scan that simulates over 50 real customer queries in just two minutes. By mapping your visibility across platforms like ChatGPT, Claude, and Perplexity, this instant audit shows where you are winning recommendations and which technical blockages are holding your brand back.