Nielsen and Hitwise: Why Are Their Search Engine Ratings Different?
Several of the search engine watchers try to estimate the distributions of the visitors to the major search engines monthly and, ultimately, to monitor how the “popularity” of these search engines changes over time. Two of the watchers are the Nielsen NetRating and Hitwise.

A close look at these graphs reveals “significant” difference between the “popularity” of the major search engines. Why are the results so different?

Danny Sullivan posted two articles on August 23, 2005, showing the distributions based on the July 2005 data collected by three two organizations. The following figures are taken from his articles:

Fig 1 - Search engine rating according to Nielsen NetRating.


Fig 2 - Search engine rating according to Hitwise.


Nielsen NetRating shows that 46.2% of the Internet surfers performed the search using Google, and at the same time Hitwise shows that 43.4% of the surfers performed the search using the same Google search engine. The difference is almost 3%!

At the first glance, the difference may look insignificant, but if your web site has the traffic of, say, 100,000 visitors per day, then 3% percent difference means 3,000 visitors. Assuming your estimate of the PPC conversion rate is 10% with the average conversion of $100, then the prediction of the daily revenue will differ by $3,000 per day depending on which search engine rating is used.

The questions is: Why are these results so different?

Let’s take a look at the sampling theory and the basis for the argument. When taking a sample from a population a sampling error is induced, which results in the uncertainty of the estimated parameter due to the fact that only a portion of the population is examined. This uncertainty is measured by the so called Margin of Error (MOR). In the case of the proportion (e.g., percent of searches performed using Google), the estimate of 46.2% has the Margin of Error of approximately 0.1% at the confidence level of 95% (my assumption) and the assumed sample size of 1,000,000 surfers.

Therefore, the estimated percent of Google surfers for July of 2005 is

46.2% ± 0.1%

In other words, if all assumptions hold, the “true” percent of the surfers who used Google search engine in July of 2005 lies in the range from 46.1% and 46.3%!
But, the Hitwise data shows that only 43.4% of the Internet surfers use Google search engine. If the Nielsen NetRating and Hitwise have taken their samples from the same population (population of the Internet surfers), the estimated percentages should not be so different.

Where does the difference come from?

Some possibilities are:

1. To be representative of the population from which it is taken, a sample must be taken in some random fashion, which implies that every member of the population (a surfer, in this case) should have approximately equal chance to be selected in the sample. If that is not the case, the result, the engine “popularity” percent will be biased and probably far from the true value. How do these two organizations select a sample of surfers from the overall population?

2. Is a different sample (1,000,000 people!) taken every month? I would assume not, but I may be wrong. If not, the bias of the estimate is introduced again because the surfing pattern in the current month is influenced by the surfing pattern applied in the previous month; i.e., “my experience from the previous month influences my behavior in this month!”

3. Do these two organizations use the same search criteria for counting the number of searches? If not, the difference in the count could be significant.

For similar discussions and insights into the various search engines statistics, statistical market analysis and application of the Six Sigma methodology and tools, read my next posting.

Dr. Vladimir Crk is WebSourced's Director of Six Sigma Quality Assurance and Marketing Statistical Analysis.
Garrett French - MarketSmart Interactive | permalink | +marktd |