How to configure your Shopify robots.txt for AI search crawlers
Claude
If you are looking to capture high-intent buyers on conversational search platforms, you must ensure your Shopify storefront is fully accessible to AI crawlers. In this guide, Pendium explains how to edit your store's default robots.txt.liquid template to explicitly whitelist citation-based user-agents like GPTBot, ClaudeBot, and PerplexityBot while keeping sensitive cart and admin paths blocked. By overriding standard Shopify settings, you allow these modern AI search agents to safely index your product details, reviews, and collection taxonomies, turning generative AI platforms into direct referral traffic sources for your business in 2026.
Open any technical search engine optimization guide written before 2025, and the advice regarding AI bots is uniform: block them all. In 2026, that default stance is actively hiding your products from shoppers using ChatGPT, Perplexity, and Google AI Overviews.
Many merchants copy-pasted rules to block AI scrapers during the early wave of LLM training debates. While blocking training scrapers makes sense for media sites, applying those same blocks to your commerce catalog means you disappear from the exact platforms where buyers are making purchasing decisions.
The six AI user-agents that matter for Shopify commerce
To build a modern AI visibility strategy, you must separate generic scrapers from citation-based search crawlers. The bots that drive direct referral traffic to your storefront require explicit access to your catalog.
The following six AI user-agents are the most important actors in conversational commerce today:
- GPTBot: The primary crawler for OpenAI, used to index web content that feeds ChatGPT search and recommendation queries.
- OAI-SearchBot: A specialized search-indexing agent from OpenAI designed to retrieve real-time data for search tasks.
- ClaudeBot: Anthropic's indexer, which gathers context for Claude when users ask for product comparisons or market research.
- PerplexityBot: The retrieval utility for Perplexity, which directly affects how your store is cited in consumer shopping answers.
- Google-Extended: The token used to control whether your site's content can be used by Google Gemini and AI Overviews.
- CCBot: The Common Crawl crawler, which feeds many open-source models and foundational training sets.
When you block these specific crawlers, you receive zero citations from those platforms. An audit of 38 Shopify stores in 2025 found that 9 of them blocked at least one AI crawler without realizing it, as documented in the 2026 Shopify store audit report.
These merchants were missing out on valuable traffic simply because their firewalls or default rule sets grouped search-based AI bots with aggressive scrapers. To prevent this, you must explicitly allow search-based crawlers while maintaining blocks on bots that scrape without giving back attribution.
Creating the Shopify Liquid override file inside your theme
Shopify automatically generates a standard robots.txt file for every storefront. While this default file protects system-heavy pages like checkout and admin paths, it does not contain custom directives for modern AI user-agents.
Because Shopify manages this file dynamically, you cannot upload a static robots.txt file directly to your root directory. Instead, you must create a robots.txt.liquid template inside your theme's code editor. This method allows Shopify to update its core rules while allowing you to write custom instructions.
To create this file, follow these steps:
- From your Shopify admin, go to Online Store and click Themes.
- Find your active theme, click the three-dot icon, and select Edit code.
- In the left sidebar, find the Templates folder and click Add a new template.
- In the dropdown, select robots as the template type and click Create template.
By default, Shopify fills this new templates/robots.txt.liquid file with a loop that pulls your store's default group settings:
{%- for group in robots.default_groups -%}
{{- group.user_agent -}}
{%- for rule in group.rules -%}
{{ rule }}
{%- endfor -%}
{%- if group.sitemap != blank -%}
{{ group.sitemap }}
{%- endif -%}
{%- endfor -%}
You must keep this loop intact. It is the code that pulls Shopify's standard rules to keep search engines from indexing your customer accounts or cart checkout flows. You can verify this template structure in the Shopify developer documentation on customizing robots.txt.
Setting explicit allow rules for your catalog and educational content
Once your override file is created, you need to add directives that allow AI bots to read your main catalog. The goal is to make your primary commercial and educational paths fully accessible to friendly user-agents.
Opening the product catalog
AI shopping engines need access to product pages, collection layouts, and category feeds. Without this information, they cannot understand details like price points, stock status, or material choices.
You should make sure that paths starting with /products/ and /collections/ are wide open to the key AI bots. When ChatGPT or Perplexity crawls these pages, they parse your structural data to verify your catalog details.
If your data is disorganized or missing, it can lead to model errors. To avoid this, you should map Shopify product taxonomy to Schema.org to stop AI misclassifications, ensuring that when AI agents crawl your products, they understand exactly what you sell.
Exposing educational content and buying guides
When shoppers ask conversational questions like "What is the best running shoe for high arches?" they are not searching for a specific product name. They are looking for educational guidance.
Your blog posts, guides, and FAQ pages are the primary resources AI bots use to answer these complex informational queries. You must keep /blogs/ and /pages/ accessible so that search agents can cite your store as the authority.
You can also use this opportunity to establish your brand's narrative. To ensure your company's values are correctly understood during these comparative searches, you should format Shopify brand settings to dictate your AI search bio and logo.
If you publish a /llms.txt file to declare your store's profile to language models, you must also make sure it is not blocked by a default wildcard rule in your robots configuration.
Preserving the default blocks on transactional paths
Allowing AI bots does not mean giving them a free pass to crawl every single page on your site. Opening up your transactional paths can lead to broken shopping sessions and skewed data.
The paths you must keep sealed
You must maintain strict blocks on your transactional and customer account systems. Allowing AI bots to parse these pages leads to major search index problems.
| Target Path | Default Shopify Rule | Reason for Blocking AI Crawlers |
|---|---|---|
/cart | Disallow: /cart | Prevents bots from creating empty sessions or triggering cart abandonment emails. |
/checkout | Disallow: /checkout | Stops crawlers from accessing checkout paths and leaking placeholder session tokens. |
/admin | Disallow: /admin | Blocks attempts to scan backend systems and protects secure log-in portals. |
/search | Disallow: /search | Prevents bots from crawl-looping infinite internal search combinations. |
/*?q=* | Disallow: /*?q=* | Stops crawlers from indexing duplicate product filters and parameters. |
Keeping these paths blocked protects your store from data leaks and prevents AI agents from quoting outdated checkout prices. Standard AI bots that follow robots protocols will read your clean catalog pages and skip these transactional paths entirely.
Bypassing edge-network blocks and firewalls
Many Shopify merchants set up their robots.txt.liquid template correctly but still find themselves hidden from AI search results. The problem often lies at the network security layer.
If you use a third party security proxy like Cloudflare, your Web Application Firewall (WAF) may be blocking unrecognized automated traffic. Many edge security packages default to a defensive posture, blocking any user-agent that looks like an automation tool.
When an AI search agent tries to fetch your product catalog, it receives a 403 Forbidden error or gets stuck on a JavaScript challenge screen. Because the bot cannot bypass this network block, it assumes your store is inaccessible and excludes your products from user recommendations.
You must review your firewall event logs to verify that bots like GPTBot and PerplexityBot are on your security platform's whitelist. If your CDN blocks these verified IPs, your custom liquid rules will never have a chance to run.
The complete Shopify robots.txt.liquid configuration
To apply these rules, you can add custom blocks directly beneath your default Shopify loop. The code below keeps Shopify's dynamic rules active while adding explicit instructions for the top six AI search agents.
Open your templates/robots.txt.liquid file and update the code to match this layout:
{%- comment -%}
Shopify default groups loop - Keeps admin and cart paths protected
{%- endcomment -%}
{%- for group in robots.default_groups -%}
{{- group.user_agent -}}
{%- for rule in group.rules -%}
{{ rule }}
{%- endfor -%}
{%- if group.sitemap != blank -%}
{{ group.sitemap }}
{%- endif -%}
{%- endfor -%}
{%- comment -%}
Explicitly allow traffic-driving AI search and retrieval agents
{%- endcomment -%}
User-agent: GPTBot
Allow: /products/
Allow: /collections/
Allow: /blogs/
Allow: /pages/
Allow: /llms.txt
Disallow: /cart
Disallow: /checkout
Disallow: /admin
User-agent: OAI-SearchBot
Allow: /products/
Allow: /collections/
Allow: /blogs/
Allow: /pages/
Disallow: /cart
Disallow: /checkout
User-agent: ClaudeBot
Allow: /products/
Allow: /collections/
Allow: /blogs/
Allow: /pages/
Disallow: /cart
Disallow: /checkout
User-agent: PerplexityBot
Allow: /products/
Allow: /collections/
Allow: /blogs/
Allow: /pages/
Disallow: /cart
Disallow: /checkout
User-agent: Google-Extended
Allow: /products/
Allow: /collections/
Allow: /blogs/
Allow: /pages/
Disallow: /cart
Disallow: /checkout
User-agent: CCBot
Allow: /products/
Allow: /collections/
Allow: /blogs/
Allow: /pages/
Disallow: /cart
Disallow: /checkout
After saving this file in your Shopify code editor, check your store's root URL by adding /robots.txt in a web browser. The page should render your standard Shopify rules first, followed by the explicit allow blocks for the six search agents.
Verifying and monitoring your store's AI visibility
Once your robots template is updated, you must verify that the new settings are working. You can test your live site with a command-line tool like curl to make sure your server returns a clean 200 OK response to these specific user-agents.
Changes to your robots directives can take several days to register across different search networks. During this window, you can run a free Pendium AI Site Audit to check if your site is fully prepared for AI discovery. The audit analyzes your schema structure, crawls your public pathways, and flags any security blocks that might still be stopping AI agents from reading your products. By keeping your product catalog open and your transactional pages locked, you ensure your brand is ready to capture traffic from the next generation of search.
