White Hat Guys

Well folks Google has ended the experiment with using two indexes for its results. This use of the main index and the so called “supplemental index” has raised havoc on the net for a couple years now and created a whole new area of SEO aimed at getting pages rescued from the index “hell” of being in the supplemental listings in Google. According to Google they are ending this experiment. Read what they had to say.

From the Google Blog

In 2003, Google introduced a “supplemental index” as a way of showing more documents to users. Most webmasters will probably snicker about that statement, since supplemental docs were famous for refreshing less often and showing up in search results less often. But the supplemental index served an important purpose: it stored unusual documents that we would search in more depth for harder or more esoteric queries. For a long time, the alternative was to simply not show those documents at all, but this was always unsatisfying—ideally, we would search all of the documents all of the time, to give users the experience they expect.

This led to a major effort to rethink the entire supplemental index. We improved the crawl frequency and decoupled it from which index a document was stored in, and once these “supplementalization effects” were gone, the “supplemental result” tag itself—which only served to suggest that otherwise good documents were somehow suspect—was eliminated a few months ago. Now we’re coming to the next major milestone in the elimination of the artificial difference between indices: rather than searching some part of our index in more depth for obscure queries, we’re now searching the whole index for every query.

From a user perspective, this means that you’ll be seeing more relevant documents and a much deeper slice of the web, especially for non-English queries. For webmasters, this means that good-quality pages that were less visible in our index are more likely to come up for queries.

Hidden behind this are some truly amazing technical feats; serving this much larger of an index doesn’t happen easily, and it took several fundamental innovations to make it possible. At this point it’s safe to say that the Google search engine works like nothing else in the world. If you want to know how it actually works, you’ll have to come join Google Engineering; as usual, it’s all triple-hush-hush secrets.*

Many of you have sites that had pages slip into these results, some for no reason at all and others for perhaps good reason. Either way it now appears to be over and now your pages will be shown in the main index if they show relevant material to web searchers. Good news for all of us. Sort of a Christmas present from Google to all of us web builders.