Unable to hide old product pages indexed on Shopify robots.txt?

Easy way to do it, no apps involved.

Shopify does not offer editing of robots txt and maintains the files itself. This is troublesome for SEO but you can hide the old product pages by just adding the ‘noindex’ tag into the required page.

Sadly, as of 2021, we do not see this issue of editing robots.txt being resolved ever since it was raised. However, in order to serve Google better, Shopify store owners have found ways to address this issue.

In this blog post, we will share what is robots.txt and how to hide old product pages on Shopify robots.txt file and also how to verify if the robots.txt is updated.

Table of Contents

What is robots.txt and how it could affect my store?

Google uses robots or we can say ‘crawlers’ to find your store product pages. When your pages are crawled by the Google robots, it helps in ranking on Google. Simply put, if Google can’t find your pages, it won’t be able to show it search results.

Every store has its robots.txt file which tells Google crawlers which pages should be crawled and which shouldn’t. However, this doesn’t mean robots.txt file keeps your pages ‘not to show’ but it avoids overloading of your website/store.

Probably, your store has too many pages indexed even if you don’t think you do, must check. You will get surprised. Point is, if a search engine crawls any store, it would crawl every single page.

If your store has a lot of pages, the search engine bot/crawler will consume more time which can negatively affect your store rankings overall.

Do you know Googlebot has a crawl budget?

Crawl budget is the number of URLs a bot crawls. This budget depends on two factors. First the crawl rate limit, here’s how Google explains it:

Shopify robots.txt

Second is crawl demand and here’s what it means:

Shopify robots.txt

In our use case where we don’t want old or outdated pages to show in Google results, we can use the ‘noindex’ tag in any of the required product pages.

What is a noindex tag?

A tag which tells Google not to crawler any specific product page and usually embedded in code like this:

You may be thinking where to add this code when Shopify doesn’t allow it. Hold your horses, there is a hope. First, you need to locate your robots.txt file in the Shopify store.

Where can I find robots.txt file?

Find your store’s robots.txt file in the root directory of your store’s primary domain name.

Example:

phaedrasolutions.com/robots.txt

List of pages must be blocked by robot.txt

prettythings.com/checkout

prettythings.com/checkouts

prettythings.com/account/register

prettythings.com/account/login

prettythings.com/finishorder

prettythings.com/postreview

prettythings.com/productimage

prettythings.com/productupdates

prettythings.com/thankyou

Why is Google banning indexing my product page URLs?

The obvious reason is to prevent duplicate pages from indexing.

Oftentimes you misread the robots.txt file. There could be several instances when Shopify won’t allow inappropriate URLs.

There could be many product pages which are duplicate – just think that you have 10 tags. There are 1,025 combinations of those 10 tags. And most of those combinations are nonsensical or don’t add a lot of value — that’s why Shopify is preventing them from being indexed.

‘Noindex technique is recommended by Google’ for hiding/removing old pages

One of the best ways to hide your product pages from being indexed is using a noindex tag as we were referring to before.

1-Go to your admin dashboard and enter into themes section. Select ‘edit code’.

Shopify robots.txt

2-Choose the theme you want to edit which is theme.liquid.

Shopify robots.txt

3-Enter the following code in the head section. We used /baby-clothes’ in the category name as shown in the screenshot.

{% if product.url == ‘/products/categoryname’ %}

<meta name=”robots” content=”noindex”>

{% endif %}

Shopify robots.txt

Hit save on the top right corner of the screen.

Verify if the robots.txt is updated by using the tester tool

Finally test your robots.txt file in order to be sure that old or outdated pages are not ranking anymore. Google provides a free tool i.e. robots.txt tester as part of the Webmaster tools.

Add the updated robots.txt there and test. Remember, changes in actual code may take a day or so.

Final verdict

SEO of a Shopify store is often tough given the fact that you can’t edit the robots.txt file directly. You have to look into hacks in order to do so. However, editing Shopify robots.txt may be a daunting task if you have to hide multiple product pages. Just make a list of all the stale/duplicate product pages and convert the page names into page handles. Adding {% else if handle contains ‘new-page-handle’ %} in the head section may not be daunting if there are only a few product pages you wish to hide.

In case you are overwhelmed with the duplicate product pages and can’t handle the robots.txt file yourself, we are always happy to help.

Heading 1

Heading 2

Heading 3

Heading 4

Heading 5
Heading 6

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse varius enim in eros elementum tristique. Duis cursus, mi quis viverra ornare, eros dolor interdum nulla, ut commodo diam libero vitae erat. Aenean faucibus nibh et justo cursus id rutrum lorem imperdiet. Nunc ut sem vitae risus tristique posuere.

Text Link

Name Email Address

What’s a Rich Text element?

The rich text element allows you to create and format headings, paragraphs, blockquotes, images, and video all in one place instead of having to add and format them individually. Just double-click and easily create content. The rich text element allows you to create and format headings, paragraphs, blockquotes, images, and video all in one place instead of having to add and format them individually. Just double-click and easily create content. The rich text element allows you to create and format headings, paragraphs, blockquotes, images, and video all in one place instead of having to add and format them individually. Just double-click and easily create content.

The rich text element allows you to create and format headings, paragraphs, blockquotes, images, and video all in one place instead of having to add and format them individually. Just double-click and easily create content. The rich text element allows you to create and format headings, paragraphs, blockquotes, images, and video all in one place instead of having to add and format them individually. Just double-click and easily create content. The rich text element allows you to create and format headings, paragraphs, blockquotes, images, and video all in one place instead of having to add and format them individually. Just double-click and easily create content.

This is an h2 heading inside rich text

A rich text element can be used with static or dynamic content. For static content, just drop it into any page and begin editing. For dynamic content, add a rich text field to any collection and then connect a rich text element to that field in the settings panel. Voila!

Static and dynamic content editing

A rich text element can be used with static or dynamic content. For static content, just drop it into any page and begin editing. For dynamic content, add a rich text field to any collection and then connect a rich text element to that field in the settings panel. Voila!

Static and dynamic content editing

A rich text element can be used with static or dynamic content. For static content, just drop it into any page and begin editing. For dynamic content, add a rich text field to any collection and then connect a rich text element to that field in the settings panel. Voila!

A rich text element can be used with static or dynamic content. For static content, just drop it into any page and begin editing. For dynamic content, add a rich text field to any collection and then connect a rich text element to that field in the settings panel. Voila!

How to customize formatting for each rich text

Headings, paragraphs, blockquotes, figures, images, and figure captions can all be styled after a class is added to the rich text element using the "When inside of" nested selector system.

Related Blogs.

How to apply Shopify multiple discount codes at checkout?How to apply Shopify multiple discount codes at checkout?
People love discounts, so your customers would love them too. Don't know how to apply multiple discount code options at the checkout? Read this guide to implement more options for your online store.
How to add video to product images Shopify?How to add video to product images Shopify?
Videos are great way to engage customers and demonstrate how a product works. Are you having trouble with adding videos to product images? We have some methods to fix this issue.
How much does a Shopify expert cost? US vs Non US developersHow much does a Shopify expert cost? US vs Non US developers
Are you looking for a Shopify expert? We have a guide on hiring Shopify developers. It includes Shopify development costs across various regions like US, UK, Australia, etc.

Contact.

You have an idea? We handle all the rest

Drop us a line.

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.

USA

500 Grant Street, Downtown, Suite 2900,
Pittsburgh, PA 15219, USA

+1 (412) 475 9053

Pakistan

2 West Canal Bank Road, Lahore 54000,
Pakistan

+92 (322) 452 4442

United Kingdom

Suite A12, 56 Wood Lane, London,
W12 7SB, UK

+1 (93) 606  7875

Project inquiries

sales@phaedrasolutions.com

It’s not about ideas
It’s about making ideas happen
Let's Talk
Quick Links
Copyright 2022 - Phaedra Solutions.
All Rights Reserved