Last Updated on September 13, 2022

The option may have vanished in the new Google Search Console view, but the robots.txt tester is still alive and well!

It’s an essential part of developing websites, to monitor and change the urls that Googlebot can access.

Here is the link if you’re just looking for the tool: https://www.google.com/webmasters/tools/robots-testing-tool

robots.txt testing tool Google Search Console

What is the robots.txt tester tool?

The robots.txt tester tool shows exactly what’s in your robots.txt file.

It can be used to test improved robots.txt files, by editing the contents in the window, and testing urls in the bottom bar.

example of blocked robots.txt page in testing tool

How to submit a new robots.txt file?

To submit a new robots.txt file to Google, your first step is to download and update the file at the correct place (root of your website).

Once you have done this, verify the page is updated by visiting domain.com/robots.txt and checking your new rules are there.

After that, go back to the testing tool, and click this submit button:

submit button new robots.txt

After this, click on the Submit button at the bottom to get Google to recrawl the page:

submit pop-up robots.txt testing tool

After this you can verify that Google has refreshed the robots.txt by reloading the testing tool, and checking the timestamp at the top of the page (and of course the rules in the box should be updated).

The Use for Ecommerce Sites

The robots.txt file is hands down the best way to control the way Google crawls large ecommerce sites, preventing them from getting stuck in paginated and faceted navigation, sort and limit urls, and all other sorts of crawl black holes.

Keeping Google crawling the pages you want to rank, and not wasting budget on the pages that will never rank.

If you need help implementing the robots.txt file on your website, then contact me via email at info@matt-jackson.com or visit my SEO services page for more information.

How useful was this post?

Click on a star to rate it!

Average rating 0 / 5. Vote count: 0

No votes so far! Be the first to rate this post.

We are sorry that this post was not useful for you!

Let us improve this post!

Tell us how we can improve this post?

E-commerce SEO expert, with over 10 years of full-time experience analyzing and fixing online shopping websites. Hands-on experience with Shopify, WordPress, Opencart, Magento, and other CMS.
Need SEO help? Email me for more info, at info@matt-jackson.com

This Post Has 0 Comments

Leave a Reply

Your email address will not be published. Required fields are marked *