Indexing in Yoast SEO: What pages to show in Google’s search results

You probably want Google and other search engines to show most of your content in the search results. But most websites also have a few pages or posts that don’t need to be featured in those search results. So how do you tell Google what pages to show and which ones not to? This is where setting pages to noindex comes into play. In this post, we’ll explain more about indexing and how Yoast SEO can help you be in control of what pages will be shown in the search results.

Let’s start with a short explanation of indexing. Before a search engine like Google can rank a page or a post and show it in the search results, it needs to index it. Even before that, a crawler must discover a piece of content before evaluating whether it’s a valuable addition to its index. One of the ways a crawler discovers pages is by crawling the XML sitemap of a website. After a page has been indexed, a search engine will understand its content and be able to rank that piece of content if it fits the user’s search query best.

Which pages to show in the search results?

Determining what should or should not be indexed by crawlers tends to be hard to understand, and it’s easy to make a mistake. You wouldn’t be the first to unknowingly set a whole post type to noindex, making it unavailable to search engines. We’ve thought long and hard about this and drastically simplified this process for you in Yoast SEO. Now it all boils down to asking a straightforward question: Do you want page x to appear in search engines?

You can find the individual settings for making your content available for indexing in the corresponding parts of Yoast SEO. On Shopify and WordPress, you can find these settings under the Content Types and Categories and Tags sections of the Settings tab of Yoast SEO.

By saying Yes to the ‘Show Posts in search results’ question in the post settings, you make sure that your posts will appear in the XML sitemap and, therefore, in the search results. If you want to exclude something, let’s say you want to exclude your category pages, you can switch the toggle to No. That way, the taxonomy or post type will not appear in the XML sitemap. Because of that, it will not appear in the search results. Whenever you set something here to not appear in search engines, it will be noindexed and kept from the XML sitemap. 

Read more: Why doesn’t Google index my content properly? »

Exclude individual posts

If you really don’t want a specific URL to show up in the search results, you’ll need to add a noindex, follow tag. Leaving it out of your XML sitemap doesn’t automatically mean Google won’t index the URL. If Google can find it by following links, Google can index the URL.

You can also exclude individual posts from the XML sitemaps with the Yoast SEO sidebar in the post editor. Or, if you’re on Shopify, you can exclude individual products with the Yoast SEO sidebar. Click on the Advanced tab and select No to the ‘Allow search engines to show this Post in search results?’ question.

View your XML sitemap

When you use Yoast SEO on WordPress, our plugin automatically generates a thorough XML sitemap for your website. Making it easy for search engines to discover all the pages you want them to discover. You can check your sitemap to see if the content you want to include appears in the XML sitemap. While there, you should check if the content you want to exclude from the sitemap doesn’t appear.

You can find your XML sitemap in Yoast SEO by going to General > Site Features > APIs > XML Sitemaps. You can click on ‘View the XML sitemap’, which will take you to your website’s XML sitemap.

We’ve eliminated a lot of the confusion around indexing content and XML sitemaps by simplifying things in Yoast SEO. But, most importantly, it is now so much easier to tell search engines what you would and would not like them to show in the search results.

More on XML sitemaps

XML sitemaps are a kind of treasure map for search engine robots. They crawl them to discover new or updated content on your site. Every site benefits from a sitemap, so Yoast SEO automatically generates a robust XML sitemap for you. Your rankings won’t soar if you add one, but it does help the crawlers to discover your content that much easier. Need more information about the use of XML sitemaps on your site? Then we have some further reading for you:

• What is an XML sitemap and why should you have one?
• The sense and nonsense of an XML sitemap
• Yoast SEO XML sitemap insights for developers

Become a Yoast SEO pro series