How to configure Shopify robots.txt for AI bots safely

To earn citations and recommendations from engines like ChatGPT and Claude, AI platforms must be able to crawl your e-commerce store's public catalog. Many e-commerce merchants do not realize that default configurations or firewall adjustments often block these modern crawlers, causing storefronts to completely miss out on critical recommendation traffic. By utilizing the Pendium AI visibility platform alongside a customized robots.txt.liquid file, Shopify store owners can safely permit access to the six major AI user-agents—including GPTBot and ClaudeBot—while maintaining rigid blocks on cart, checkout, and private customer pages. This technical setup ensures your e-commerce catalog remains crawlable for real-time shopping recommendations in 2026 while keeping aggressive scrapers away from your backend storefront files.

What Shopify default robots.txt configurations do to AI crawlers

Shopify auto-generates a standard robots.txt file at the root domain of every e-commerce storefront. This file is designed to guide traditional search engine indexers like Googlebot and Bingbot while shielding sensitive system paths from search engine results pages. By default, Shopify writes rules to block scrapers from indexation paths like /admin, /cart, /checkout, /orders, and various internal search filters.

For unrecognized bots, Shopify applies the wildcard group User-agent: *, which theoretically allows access to all public catalog files. Under normal conditions, this wildcard access allows search bots to read your collections, product listings, blogs, and standard pages. The default structure works well for traditional search indexing, but it introduces massive blind spots in the age of generative search.

The first issue stems from third-party security layers and content delivery networks. When e-commerce brands implement aggressive firewalls like Cloudflare or use third-party security applications to protect their inventory from scrapers, these filters often catch new AI crawlers. Because these crawlers are unrecognized, firewalls treat them as suspicious user-agents and block them with a 403 challenge screen.

Our team's assessments at Pendium show that a significant portion of Shopify storefronts suffer from these hidden blocks. The bots log the security challenge, classify the page as inaccessible, and stop trying to index the catalog for days. If you want to identify where your storefront stands before editing your templates, you can run an audit using our guide to how to audit your Shopify catalog for AI search blockers.

The six AI bots to explicitly allow on your Shopify store

To ensure your brand appears in conversational search results on platforms like Perplexity, Gemini, and ChatGPT Search, you must explicitly identify and permit the correct user-agents. The AI visibility platform tracking engine monitors crawling behavior across seven core systems to evaluate how recommendations are formed.

To make your catalog crawlable, you should explicitly define permissions for the following six crawlers.

As documented in Shopify Robots.txt for AI Crawlers (GPTBot, ClaudeBot, 2026), a strategic divide exists between training bots and search bots. Some merchants choose to block training bots like GPTBot while permitting search-specific bots like OAI-SearchBot.

Blocking training-focused bots stops companies from using your storefront data to train future models, while allowing search bots ensures your products can still be cited in active user searches. Managing this selective control requires a custom Liquid template that defines rules on a per-agent basis.

Creating the robots.txt.liquid override file

Shopify merchants cannot edit the root text file directly from the administration dashboard. Instead, you must create a custom theme template file called robots.txt.liquid that tells Shopify how to render the final /robots.txt output dynamically.

Before customizing this template, you can run a technical crawl analysis using the Pendium AI Site Audit tool. This audit will identify whether any collection files or structured schema layouts are currently blocked from automated crawlers.

Locating the templates folder

To create the override file, log in to your Shopify admin dashboard and navigate to Online Store > Themes. Click the three-dot icon next to your active theme and select Edit Code to open the layout editor.

In the left sidebar, locate the Templates folder, right-click it, and select New File. In the dropdown menus, set the template type to "robots" and name the file robots.txt.liquid as outlined in the Shopify Help Center's robots.txt guide.

Preserving the default Liquid loop

The newly created file will populate with Shopify's default routing loop. It is critical that you do not delete this code block to write plain text rules. The default loop is structured as follows:

{% for group in robots.default_groups %}
  {{- group.user_agent -}}
  {% for rule in group.rules %}
    {{- rule -}}
  {% endfor %}
  {%- if group.sitemap != blank -%}
    {{ group.sitemap }}
  {%- endif -%}
{% endfor %}

This Liquid block dynamically fetches Shopify's backend security rules and site routing paths. If Shopify updates its core platform security, this loop automatically pulls the updated rules into your live robots file.

Replacing this loop with static text permanently disconnects your storefront from automated updates, which can lead to accidental indexation drops during subsequent system upgrades. For merchants seeking to adjust their theme templates safely, keeping crawler access operational is a major priority, as outlined in our guide on protecting your ChatGPT and Claude traffic during a Shopify migration.

Adding safe allow rules for your catalog

Once you have secured the default Liquid loop in your theme editor, you can append your custom rules directly below the closing loop tag. This approach maintains Shopify's default security rules while introducing explicit instructions for AI search agents.

Targeting the product catalog

To give AI crawlers access to your product catalog without exposing administrative pages, write explicit allow directives targeting public folder paths. This tells the crawling systems exactly where your structured product data is stored while preventing them from wasting crawl budget on system assets.

Add the following rules to the bottom of your robots.txt.liquid template:

# Explicitly allow primary AI search bots to index the public catalog
User-agent: OAI-SearchBot
Allow: /products/
Allow: /collections/
Allow: /blogs/
Allow: /pages/

User-agent: GPTBot
Allow: /products/
Allow: /collections/
Allow: /blogs/
Allow: /pages/

User-agent: ClaudeBot
Allow: /products/
Allow: /collections/
Allow: /blogs/
Allow: /pages/

User-agent: PerplexityBot
Allow: /products/
Allow: /collections/
Allow: /blogs/
Allow: /pages/

User-agent: Google-Extended
Allow: /products/
Allow: /collections/
Allow: /blogs/
Allow: /pages/

Writing these explicit blocks overrides the generic restrictions that security applications or custom firewalls might apply to unknown crawlers. This specific design follows the theme structure details outlined in the Shopify developer documentation on robots templates.

Maintaining blocks on transactional pages

While you want AI engines to read your collections and blog resources, they should never access checkout, shopping cart, or login folders. Allowing crawlers to parse transactional pages wastes your server's crawl budget and can trigger security anomalies.

The default group loop handles transactional security automatically, but you must avoid writing broad wildcard allow rules like Allow: / for AI bots. If you write Allow: / for a user-agent, it can override Shopify's automated system blocks, exposing cart folders and user directories to crawlers. Keeping your rules explicitly scoped to public-facing folders ensures your administrative paths remain protected.

Verifying e-commerce crawler accessibility and performance

Saving your updated template deploys the new rules instantly to your /robots.txt path. However, you must verify that external firewalls and content delivery networks are not silently blocking these bots before they even reach your Shopify theme.

You can verify that your new rules are working correctly by executing a simple terminal test. Open your command line interface and run the following command to simulate a crawl request from the OpenAI search crawler:

curl -I -A "OAI-SearchBot" https://yourstore.com/products/example-product

If the command returns a standard 200 OK header response, your storefront is permitting access correctly. If the command returns a 403 Forbidden error or redirects to a security challenge screen, your firewall or CDN is blocking the crawler before it can read your robots file.

To fix this, log in to your CDN dashboard (such as Cloudflare) and create a custom Web Application Firewall (WAF) rule. Set the rule to bypass bot verification checks for incoming user-agents matching the six AI crawlers listed in your robots file.

Once you have verified that AI agents can crawl your public catalog safely, the next step is measuring how effectively they interpret your product information. Run a free visibility scan at Pendium's website to see how ChatGPT, Claude, and Gemini currently perceive your e-commerce brand.