Hi, I’m Kate. In this tutorial, we will go over changes you can make to increase the chances of your content being indexed. Use the timestamps in the description to jump to specific measures you can take. First are the technical aspects, then improvements you can make to your content. Before we start, we need to understand the difference between crawling and indexing.
Google’s search bots spend all day visiting web pages, analyzing their content, and following the links in them to other pages. This is called crawling. Then Google stores that page in cache and analyzes the page to define if it will be added to its index, as well as what position it will be placed.
This may take days, weeks or the page may not be indexed at all. At the end of the day, it’s up to Google, but there are steps we can take in technical SEO, as well as content creation, to increase the chances of pages being indexed.
Let’s start with the technical: first, let’s check if your webpage isn’t actually indexed, or just isn’t ranking for the search terms you’re trying. Enter the URL of the page you want to check in Google search. If it appears, Google has already added your page to its index. If the page doesn’t appear, you need to check if your posts, pages or entire website had the Noindex robots meta tag in them. You can start by looking for the tag in the page source code, with the keyboard shortcut Ctrl + F or Command + F on a Mac.
Or use Rank Math’s SEO Analyzer free tool, which we left a link to in the description. In the Advanced SEO section, the Noindex Meta test lets you know if the tag is present or not.
If it is present, open the post or page in the editor and access Rank Math’s Metabox. In the Classic Editor, it’s right below the content box where you write your posts. In Gutenberg, you’ll need to click on the button with Rank Math’s logo and the post’s SEO score.
Go to the second page, Advanced, and check the Index box. Don’t forget to save the changes. However, you might notice that several pages, posts, or items from a custom post type aren’t indexed. In that case, open Rank Math’s Titles & Meta settings and go to the tab of the post type. Activate the Post Robots Meta option and select the Index option.
Save the changes at the bottom of the screen. If you notice that both pages and posts haven’t been indexed, there might be something preventing Google from indexing your entire website. Let’s take a look at WordPress settings. In the admin area, go to Settings, Reading, and ensure that the option “Discourage search engines from indexing this website” is not enabled. This might be active because a lot of web developers use it to prevent Google from indexing a page during the development process, but forget to remove it before publishing.
The last test you can do from within your website is looking for a Disallow tag in your website’s robots.txt file. This blocks robots from crawling all the pages on your site. Watch the video on the card to learn how to work with the robots.txt file.
You can try to learn more about why a particular page wasn’t indexed with Google Search Console’s URL inspection tool. If it says “Crawled – currently not indexed”, Google has crawled your URL but decided not to index it. Now that we’ve covered the practical steps that allow your site to be indexed, let’s see what you can do to increase the chances of that happening. Google does not disclose anything specific that they look for while indexing, but you can consider some well-known quality factors. One of them is the internal linking structure: if you’re trying to index an important page, make sure it has internal links from relevant pages on your website.
To start building internal links, you can set your important pages as Pillar Content and let Rank Math suggest relevant internal links. Another factor is content duplication. If two or more pages on your website are about the same topic and you can’t combine them, you need to add a canonical URL for these duplicate pages, pointing to the one that you want to index. In case you’ve never heard of canonical tags, they tell search engines which URL in the website should be indexed for the content on that page. They can point to another URL or the page itself.
Use them so that your website doesn’t compete against itself for ranking.
Plagiarized content also stops a page from being indexed. Google usually indexes posts within a week or two once the copies content is removed. Other factors include the frequency of posting and your website’s domain authority, among many more. All of these aspects, especially the last two, affect the crawl budget that Google assigns to your website.
When we say budget, we are talking about the amount of time that search bots will spend crawling on your website. After all, keeping the search bots at work costs Google money. Once you’ve made changes to the indexing settings or the content, you can wait for Google to index your website, which could take a few weeks, but most of the time happens much quicker. If you’re in a hurry, use Rank Math’s Instant Indexing plugin to take advantage of the Google Instant Indexing API and have that page crawled in minutes. We made a tutorial about this plugin.
It’s the video on the card. If this tutorial helped you, give it like. If you have any questions, click on the link in the description to open a free support ticket.
You can find more tutorials in our knowledge base, at rankmath.com/kb/.