If you’ve found this post then welcome to the Google’s choosing not to index my content club!
From my own experience - and plenty of other voices on SEO Twitter - we’re collectively seeing an increase in the number of URL’s which should be indexed, but for one reason and another aren’t.
It’s been an increasing issue for me these past few months as more and more content on this site is forever stuck in the ‘Crawled - currently not indexed” coverage report in Google Search Console. Even my service page on SEO Audits took over 6 weeks to be indexed. This has resulted in a fair bit of frustration as I appear I thought I’d been doing all the right things.
MY CRAWLED - NOT CURRENTLY INDEXED CHECKLIST
Are there any technical bugs and/or glitches?
Is the XML Sitemap working?
Is Robots.txt working?
Are onpage elements configured correctly?
What is the content quality of the page?
Are there any duplicate content issues?
What is the quality of the content hub/website?
What is the intent of the content hub/website?
Does the website show strong Google EAT signals?
What is the quality of the domain’s backlink profile?
WHY IS MY CONTENT STUCK IN ‘CRAWLED - NOT CURRENTLY INDEXED’ PURGATORY
Putting emotions to one side, I’ve tried to be objective and work through a range of different items which may be causing the issue. So, what follows is a bit of a brain dump combining things I’m testing, tweets and conversations with other SEO’s.
TECHNICAL BUGS AND GLITCHES
My site runs on SquareSpace so first of all I inspected the URL to check Google could actually find and crawl several URL’s. Thankfully the mounds of code which makes Squarespace, well Squarespace, is all running fine so no issues there.
XML SITEMAP
Was the URL included in the XML sitemap? Yes it was (with Squarespace this is automatic)
ROBOTS.TXT FILE
Could the website unwittingly be blocking the URL from being crawled? In this instance you can’t edit the robots.txt file and as mentioned inspecting it in Google Search Console meant the big-G could still locate the content I wanted it to discover, crawl and index.
ONPAGE ELEMENTS
Were all the key elements which make up a page all present and correct? We’re looking at the title tag, description, headings, images, alt-text and so on. Answer - all present and correct.
CONTENT QUALITY
Lastly, was the page any good? In other words was it original, of a decent length and did it satisfy the intent of someone searching for ‘SEO Audit consultant’. In order to make the page I looked through plenty of other ranking URL’s in the SERP to understand what they talked about, how much they wrote and so on. So the page I created essentially ticked every box.
EXPANDING OUTWARDS
At this stage we start to expand things out a little further than beyond just a single URL and tried to understand a little more about it’s relationship to the rest of the site…
DUPLICATE CONTENT
Have I inadvertently duplicated the post, either because I’ve:
Copied and pasted the copy elsewhere on the site
Forgotten I had already used the page content to feature on another site
Due to a technical mishap with tags and categories and an incorrect configuration of canonical tags multiplied the page content across several URL’s?
In this instance, no, no and no. Although these all seems like likely reasons to get a page kept out of the index.
INTERNAL LINKING
I’m not even going to ask the question, but yes there were a number of internal links added - in this instance from the main navigation, other service pages and deeper links within other blog posts for Google to find - and yet here we are.
At this stage I was stumped and was undertaking some serious head-scratching. All of the essential items were added in the same way I have done so for many years - and yet no bite from Google. Slightly more drastic action was required.
CONTENT QUALITY OF THE CONTENT HUB/WEBSITE
If I was to look at my URL within the context of the wider website, what is the quality of all the other posts and pages on there? The answer for me was - not good. Technically it was all sound but I feel like my blog could have been causing issues. I’ve been on Squarespace (and blogging) for 9 years no and while not a furious copywriter have amassed 50+ posts which covered all aspects of life, marketing and other stuff.
Over several months I’d seen a steady increase in the number of blog URL’s added to the ‘Crawled - but not indexed’ pile. I can’t say I was surprised. A lot of the content was incredibly thin, out of date and nobody was finding it, reading it or doing anything with it. However, for sentimental reasons I wanted to keep it as it documented my progress in my career since graduation through to where I am today. It had to go.
As it represented such a large percentage of the total number of pages on the website I knew it wouldn’t be doing me any more favours. So each piece has been quietly archived and the URL redirected to clear things out. I have kept a few choice pieces - like how I built a website to win my 2nd job - but overall what was 50 pieces is now down to about 8.
WEBSITE INTENT
There seems to be an unwritten rule that a content hub/website with a laser focus intent seems to perform better as a whole than a website which is all over the place when it comes to the topics it talks about. I’ve seen this in action with The Way With Words where our site which an incredibly few inbound links and only 14 posts all about tone of voice outranks some well recognised names.
Applying this to my site though, is everything on here all about SEO?
It is now - but just a few months ago that wasn’t the case.
Since setting up my own business I’ve offered a complete range of digital marketing services. Originally it was everything, then quickly became a core of social media, content marketing, PPC and SEO. Social was then dropped in favour or ‘search marketing’ covering SEO, PPC and content. Then in Winter 2021 PPC was dropped and here is where you find the website today. SEO with content marketing (also for SEO). Returning to the blog the posts that I’ve added throughout the years reflect these changes. So there’s been posts about social media listening, freelance life and a couple of posts on my interests. Hopefully interesting - but nothing to do with SEO.
After the grand-clear out though I’ve been left with a much slimmed down website but still not many signals which shout SEO to either readers or Google.
Since the start of the year that’s all changed. The goal for 2022 has been to write something once a month, every month and so far I’ve been succeeding. In it’s place I’ve been making a conscious effort to write up detailed, original and hopefully helpful posts about SEO - and hoping that by taking the old out and bringing in the new it’ll have a positive impact. The latest posts include:
and that leads me onto…
EXPANDING OUT FURTHER
GOOGLE EAT
We know about Google EAT. We know about Google’s Quality Rater Guidelines. We know that someone selling their marketing services is going to fall somewhere in YMYL territory.
Across my site and socials there’s definitely enough to hopefully show that I’m an Expert and hopefully Trustworthy. Am I an authority? Not really.
While there’s author links across this site and occasional glimpses of my thoughts and opinions elsewhere online there isn’t really anything substantial which adds up enough to say that I’m really doing well within the EAT part of the algorithm, especially when the marketplace for SEO is crowded and competitive.
This then leads me onto:
LINKS
I need to be doing more on other websites. I need to be contributing, sharing and talking about what I know about on other sites, at other events and generally making a name for myself.
As well as the external signals that must feed into all things Google EAT it’ll mean some additional relevant external links coming back to this site. As we all know links are a huge component part of increasing authority and ranking position. And the backlink profile for this site is small, old and generally not as good as it can be. So there’s definitely room for improvement.
WRAPPING UP
After around 6 weeks of waiting something finally persuaded Google to index by very important SEO audit page. What it was I can’t say but looking back through the things I’ve worked through there’s definitely a common theme of quality. The bar for what’s included in Google’s index is so much higher - and rightly so. It’s something that I’ll continue to adapt to. The bigger conundrum though is how to explain to clients with an outdated understanding of SEO for how they’re going to do the same.