It’s been about a month since I posted about Robots.txt and how it can help you escape supplemental hell, so I thought it was time for a quick update.
At the time of writing, I had around 250 pages in Google’s main index (around the number of posts and pages at that time), with 550 pages in the supplemental index, which as far as I can determine is not a good thing: you should have as many pages as possible in the main index.
Since incorporating the robots.txt file as outlined in the original post, a search of the 2 Google indexes now reveals Blog-Op to have 431 pages in the main index, and only 49 in the supplemental, which is quite a turnaround.
I can’t say my Google traffic has exploded as a result, but I probably get a steady 40-50 hits a day out of Google visitors, so one thing I can say with certainty is that it hasn’t hurt my blog. If you have a large number of pages in the supplemental index that you’d prefer in the main index, then you may want to give this a try as it appears to work.
I’ve had similar results for my other 2 blogs I’ve incorporated the robots.txt on, and I’ll continue to keep you updated with my experiences. To find out your supplemental stats, and what my robots file looks like, please check out the original post.
Has anyone else tried this?
5 users commented in " Robots.txt & The Google Supplemental Index-Update "
Follow-up comment rss or Leave a TrackbackI’ve whittled my supplimental results down from around 350 to 26. And I’ve made a few more tweaks to my robots.txt file that may reduce this a smidge more.
To avoid the duplicate content from counting against me, should I exclude *.xml files so Google isn’t picking up my feed files? Or will this cause me grief with my sitemap.xml file?
That’s pretty good Rob.
My feed files are about the only ones left in the supplemental as well, and I’m tempted to say that excluding the XML files would mess up accessing your Sitemap.
From Andy Beards comments on my earlier post, supplemental isn’t the end of the world, and I would suggest that getting the bulk of your posts from the supplemental to the main index is enough.
Let us know if you decide to experiment though
Hey Chris,
just thought I’d let you know I’m selling my blog!
All the details are on the homepage.
Matt
I have implemented this before, it works but it takes some time to clear out you pages from supplemental index. From my experience, it clear out almost 7k URL in 2 weeks.
Other than robots.txt, you can try using google webmaster removal URL tool
Thanks Zaki, good tip.
Leave A Reply