When you start a new website, the first that you are going to be worrying about in terms of search engines is whether or not they have your site indexed. If they don’t, then you are going to stand no chance of getting any further in terms of SERP position, no matter how much SEO you throw at your website. Ensuring that your site is indexed is simple but often, people will forget about this process and not even realise that their site isn’t indexed. There are a number of reasons that Google, and other major search engines might choose not to index your website and by knowing what these are, you can ensure that you overcome any problems.
#1 – Errors in robots.txt
The robots.txt file is something that should be present with all websites. It only takes a few minutes to produce in most cases and it will drastically help when it comes to indexing your website. Basically, the robots.txt file is a file that tells search engines where they are allowed to look on your website. It tells then what can and can’t be indexed. This is the first place that a search engine spider will look when they visit your website so you need to ensure that there are no errors in here.
The thing about a robots.txt file is that you can actually specify whether search engines index your site, and/or certain sections at all. Often, when you are creating a website, you will set this to disallow search engines from indexing your site as it won’t yet be complete. One of the most common mistakes here is that people forget about this when they actually do launch their website. This means that search engines are still not being able to look at your website, despite the fact that you have no finished it.
The way that you can check if this is the case is to look at your robots.txt file and see if it contains the line “User-agent: * Disallow /”. This means that search engine indexing is not allowed for any of your website. Remove this line if it’s there.
#2 – A nofollow/noindex meta tag
Something else that can often lead to your website not being indexed is the inclusion of the nofollow attribute in your meta tag. Again, this is a common mistake and it will mean that the search engine is not allowed by yourself to index or follow the page.
This is a pretty easy problem to sort out, you just need to do a search for this meta tag across all of your sites content or if you are using WordPress or another similar CMS, there is more than likely a plugin for this.
#3 – 404 errors
Again, this is another big problem that contributes to reasons that your content might not be indexed. Often, when you have a website, you might choose to move content to another area of your website for a number of reasons. This is not good for search engines as they are going to get an error when they try and find your content. This means that your content will not be indexed and with the latest updates to the Google Algorithm, this is probably going to slightly affect the whole of your site in terms of SEO as well as it shows that your content is not being cared for.
The way to fix this problem is to always ensure that you 301 redirect any moved content. If you use WordPress, this will automatically be done for you in most cases but if you have no CMS or a different CMS, you need to check that it is doing this. You should also check that it is not a 302 redirect that is being used as this will not pass any link juice and therefore be poor in terms of SEO.
#4 – Out of date sitemaps
One issue that a lot of people will actually run into is that they do not update their sitemaps. If you have a sitemap that is generated with a WordPress plugin, then this might not be the case as it is often automatically updated. However, if not, you will need to ensure that it is updated when you add content to your website. If it isn’t, the search engines might not know your new content is there and therefore, not index it.
Adding things to your sitemap is easy but if you are adding a lot of content to your website on a regular basis, you definitely want to make sure that you have a WordPress plug-in taking care of this as it will be a time consuming process that can actually be completely automated.
#5 – A poor internal linking structure
This one is a little bit more to do with the overall SEO of your website and content pages but it is quite important for indexing too. Often, you can overcome an indexing problem with the correct sitemap, but still, sometimes, Google and other search engines might choose not to index deep content if it is really lost within your website.
This is very rare but can cause indexing problems so you should be aware of it. A good internal linking structure is paramount to the success of a website anyway so make sure that you have it covered.
Getting your site indexed as well as all of the internal content is just stage one of getting a site to work. Once indexed, you need to work on the position of your content in the search results for your targeted keywords. However, indexing is step one of the equation and something that a lot of people experience issues with. Hopefully, this article has cleared up some of the reasons why content doesn’t always get indexed.