Shopify does not offer editing of robots txt and maintains the files itself. This is troublesome for SEO but you can hide the old product pages by just adding the ‘noindex’ tag into the required page.
Sadly, as of 2021, we do not see this issue of editing robots.txt being resolved ever since it was raised. However, in order to serve Google better, Shopify store owners have found ways to address this issue.
In this blog post, we will share what is robots.txt and how to hide old product pages on Shopify robots.txt file and also how to verify if the robots.txt is updated.
Table of Contents
What is robots.txt and how it could affect my store?
Google uses robots or we can say ‘crawlers’ to find your store product pages. When your pages are crawled by the Google robots, it helps in ranking on Google. Simply put, if Google can’t find your pages, it won’t be able to show it search results.
Every store has its robots.txt file which tells Google crawlers which pages should be crawled and which shouldn’t. However, this doesn’t mean robots.txt file keeps your pages ‘not to show’ but it avoids overloading of your website/store.
Probably, your store has too many pages indexed even if you don’t think you do, must check. You will get surprised. Point is, if a search engine crawls any store, it would crawl every single page.
If your store has a lot of pages, the search engine bot/crawler will consume more time which can negatively affect your store rankings overall.
Do you know Googlebot has a crawl budget?
Crawl budget is the number of URLs a bot crawls. This budget depends on two factors. First the crawl rate limit, here’s how Google explains it:
Second is crawl demand and here’s what it means:
In our use case where we don’t want old or outdated pages to show in Google results, we can use the ‘noindex’ tag in any of the required product pages.
What is a noindex tag?
A tag which tells Google not to crawler any specific product page and usually embedded in code like this:
You may be thinking where to add this code when Shopify doesn’t allow it. Hold your horses, there is a hope. First, you need to locate your robots.txt file in the Shopify store.
Where can I find robots.txt file?
Find your store’s robots.txt file in the root directory of your store’s primary domain name.
Example:
phaedrasolutions.com/robots.txt
List of pages must be blocked by robot.txt
prettythings.com/checkout
prettythings.com/checkouts
prettythings.com/account/register
prettythings.com/account/login
prettythings.com/finishorder
prettythings.com/postreview
prettythings.com/productimage
prettythings.com/productupdates
prettythings.com/thankyou
Why is Google banning indexing my product page URLs?
The obvious reason is to prevent duplicate pages from indexing.
Oftentimes you misread the robots.txt file. There could be several instances when Shopify won’t allow inappropriate URLs.
There could be many product pages which are duplicate – just think that you have 10 tags. There are 1,025 combinations of those 10 tags. And most of those combinations are nonsensical or don’t add a lot of value — that’s why Shopify is preventing them from being indexed.
‘Noindex technique is recommended by Google’ for hiding/removing old pages
One of the best ways to hide your product pages from being indexed is using a noindex tag as we were referring to before.
1-Go to your admin dashboard and enter into themes section. Select ‘edit code’.
2-Choose the theme you want to edit which is theme.liquid.
3-Enter the following code in the head section. We used /baby-clothes’ in the category name as shown in the screenshot.
{% if product.url == ‘/products/categoryname’ %}
<meta name=”robots” content=”noindex”>
{% endif %}
Hit save on the top right corner of the screen.
Verify if the robots.txt is updated by using the tester tool
Finally test your robots.txt file in order to be sure that old or outdated pages are not ranking anymore. Google provides a free tool i.e. robots.txt tester as part of the Webmaster tools.
Add the updated robots.txt there and test. Remember, changes in actual code may take a day or so.
Final verdict
SEO of a Shopify store is often tough given the fact that you can’t edit the robots.txt file directly. You have to look into hacks in order to do so. However, editing Shopify robots.txt may be a daunting task if you have to hide multiple product pages. Just make a list of all the stale/duplicate product pages and convert the page names into page handles. Adding {% else if handle contains ‘new-page-handle’ %} in the head section may not be daunting if there are only a few product pages you wish to hide.
In case you are overwhelmed with the duplicate product pages and can’t handle the robots.txt file yourself, we are always happy to help.
