Why hiding your Shopify 'all' collection blocks AI search indexing

When Shopify merchants attempt to clean up duplicate content warnings in Google Search Console by hiding their auto-generated /collections/all page, they often accidentally block AI assistants from crawling their product inventories. Modern AI engines like ChatGPT and Perplexity rely on these comprehensive collection pages to construct a complete map of your store catalog. Our testing at Pendium, an AI visibility platform, reveals that applying a blanket noindex or restrictive robots.txt rule to this directory completely blinds search bots like OAI-SearchBot and PerplexityBot. The correct solution requires creating a custom manual collection assigned to the "all" handle, configuring precise liquid conditional rules in your layout template, and auditing crawler access to differentiate between model training and live citation queries.

The problem: SEO fixes that break AI visibility

For years, technical SEO consultants have recommended hiding Shopify's default product listing page. This auto-generated directory, accessible at yourstore.com/collections/all, pulls every active product into a single, massive archive. When Googlebot encounters this page, it crawls page after page of paginated results, wasting your crawl budget and leading to duplicate content flags in Google Search Console.

To resolve this, many store owners implement quick fixes to hide the page from indexing altogether. They look at the endless pagination and add code snippets to their template or use third-party apps to block the directory. This classic SEO cleanup tactic is documented widely, such as in this active Shopify Community thread where merchants swap snippets to keep the page out of Google's index.

However, what works for Google can cause a silent crisis for your store's performance on generative engines. When you hide or block this specific collection, you remove the primary index that AI search agents use to understand your entire inventory. At Pendium, we monitor how these decisions directly impact brand recommendations, and we frequently see stores drop to zero AI visibility because of a simple SEO template edit.

Why it happens: how AI bots crawl Shopify differently

Traditional search engines and modern conversational AI platforms do not read your website the same way. Google relies heavily on structured XML sitemaps to discover your pages, using them to slowly build its search database over time. AI search engines operate under different constraints, often relying on flat directories and complete collection pages to quickly understand your brand structure and match products to conversational user prompts.

When an AI search engine receives a prompt like "find me lightweight waterproof hiking backpacks with a lifetime warranty," it does not just search for keywords. It attempts to parse your entire catalog structure to evaluate product relationships and attributes. If the system cannot find a comprehensive collection page like the default "all" catalog, it struggles to verify your current inventory.

The Google Search Console panic

The rush to block the /collections/all URL stems from standard ranking fears. When Google indexes pages like /collections/all?page=2 or /collections/all?page=15, it crawls identical product listings in a different order. This creates classic duplicate content issues and dilutes your organic link equity across multiple thin pages.

Merchants panicked by these search console warnings often copy code to force a noindex tag onto any URL containing the "all" handle. The goal is to keep Google clean, but the execution is too broad. By applying a heavy-handed template block, you inadvertently tell all crawlers—including conversational search agents—to ignore your store's most comprehensive product listing.

Answer bots vs. training bots

To make matters more complex, you must understand that not all AI crawlers have the same goals. Many store owners install apps or write rules to block AI systems out of fear that their copy and images will be scraped to train large language models. But a blanket ban does more harm than good if you block the wrong user agents.

As detailed in Kaspian Fuad's guide to blocking AI crawlers on Shopify, you must split these crawlers into two distinct buckets. Training crawlers like GPTBot and CCBot pull data to train future models, which you can block with zero SEO impact. Answer crawlers like OAI-SearchBot and PerplexityBot are real-time search crawlers that fetch your pages to answer immediate user queries.

If your Shopify theme blocks all bots, or if your template serves a noindex tag to any crawler on the /collections/all directory, your products will never show up in ChatGPT or Perplexity search results. At Pendium, our data reveals that stores blocking these real-time search agents experience a massive decline in conversational discovery, making them invisible to high-intent buyers.

The solution: fixing catalog visibility without hurting SEO

You do not have to choose between satisfying Google and being visible to AI assistants. It is entirely possible to clean up your search console errors while keeping your catalog fully readable for conversational search agents. The fix requires moving away from crude, store-wide blocks and implementing a more precise technical configuration.

Using an AI visibility platform like Pendium allows you to audit these structural patterns easily. By testing your theme's rendering and crawlability, you can verify whether your collection layout is readable for AI agents. Let us look at the exact steps required to resolve this configuration issue on your Shopify store.

Create a custom "all" collection

The default Shopify "all" collection is an automated directory that you cannot edit or control directly in the admin dashboard. However, you can override this default behavior by manually creating a new collection and assigning it the exact handle all.

To do this, go to your Shopify admin, create a collection, and name it "All Products" or "All." In the search engine listing preview at the bottom of the page, edit the URL handle to ensure it reads exactly all. You can then set this collection to pull in your products based on specific rules, allowing you to exclude out-of-stock items, draft products, or gift cards that clutter your catalog.

This manual override gives you control over the collection's metadata, description, and layout. It also allows you to integrate advanced search utilities, such as configuring Algolia search embeds on your custom all collection to ensure both human users and AI crawlers find a fast, clean, and highly organized product directory.

Update theme.liquid logic

Once your custom collection is live, you must apply a conditional indexing rule in your theme code that specifically targets Googlebot while leaving conversational agents unblocked. Instead of using a simple, blanket noindex tag for the entire collection handle, you can write liquid code that checks the incoming user-agent.

Open your theme.liquid file and locate the <head> section. You can replace your old, restrictive noindex code with a snippet that checks if the crawler is a known AI search bot before serving the block instruction:

{% if template contains 'collection' and collection.handle == 'all' %}
  {% unless req_user_agent contains 'OAI-SearchBot' or req_user_agent contains 'PerplexityBot' %}
    <meta name="robots" content="noindex, follow">
  {% endunless %}
{% endif %}

This code tells traditional search engines like Googlebot not to index the massive /collections/all directory, preventing duplicate content issues. At the same time, it ensures that search bots like OAI-SearchBot can still fetch the directory to learn about your products. To confirm if your code edits are working as intended, you can run a quick check using the Pendium AI Site Audit tool to see if your pages are readable for AI agents.

Audit your robots.txt.liquid

The final step is to ensure that your store's robots.txt file is not blocking these crawlers before they even reach your HTML templates. Shopify uses a template called robots.txt.liquid to generate this file automatically. If your theme has a legacy robots.txt override that blocks all automated agents, no template-level edits will work.

You should verify that your robots.txt file explicitly allows conversational bots to crawl your /collections and /products paths. A typical configuration should look like this:

User-agent: OAI-SearchBot
Allow: /collections/all
Allow: /products/

User-agent: PerplexityBot
Allow: /collections/all
Allow: /products/

By ensuring these rules are present, you guarantee that search agents can access your primary product directories. This allows the bots to fetch your catalog schema, product pricing, and inventory levels, which are critical for matching your store with relevant user recommendations.

When it's more serious

The issue of catalog visibility becomes even more complicated when your store relies on custom tech stacks, headless architectures, or heavy page-builder apps. Many third-party design apps rewrite your collection templates entirely, loading product grids through asynchronous JavaScript that search crawlers cannot easily parse.

When you use drag-and-drop page builders to design your collection pages, they often package your products inside complex scripts or nested frames. If an AI search engine attempts to fetch your /collections/all page and encounters a blank screen or a loading spinner, it cannot index your inventory. This is a common issue we analyze at Pendium, where visually striking stores remain completely invisible to modern search agents.

To understand how these layout applications impact your search presence, you can read our deep dive on why Shopify page builders block AI crawlers. Often, security apps or Cloudflare configurations designed to prevent scrapers end up misidentifying conversational search agents as malicious bots, blocking their access entirely and removing your products from generative shopping results.

Prevention

Because Shopify themes and application ecosystems update constantly, catalog discoverability is not a one-time fix. A routine theme update, a new SEO app installation, or an adjustment to your firewall settings can easily overwrite your custom robots.txt.liquid or theme.liquid code, making your store invisible to AI search overnight.

To prevent these silent visibility drops, marketing teams must establish clear protocols for monitoring how AI engines perceive their store. Relying on manual checks is inefficient, as you cannot realistically simulate how dozens of different search agents and customer personas interact with your site on a daily basis.

This is where a dedicated monitoring strategy becomes essential. By using the Pendium visibility monitoring dashboard, you can track your brand's search performance across major platforms including ChatGPT, Gemini, Claude, and Perplexity. The platform runs continuous checks against your store's structural health, alerting you the moment a code change or security update blocks an AI search bot from reading your catalog.

If you want to ensure your Shopify catalog is fully readable and ready to be recommended by modern conversational search engines, take action today. Visit the Pendium website to run a free AI Visibility Scan on your store's URL. In just two minutes, you will receive a complete analysis of how major AI platforms perceive your brand, helping you identify and fix the silent technical blockers that are costing you high-intent customers.