We carried out the experiment on August 2, 2002. All 58 queries were submitted to Google, AltaVista, AlltheWeb and WiseNut. The top 50 results from each search engine were retrieved for further analysis according to the principles described in last section. Here, we give our main results of the experiment.
After analyzing the results returned by each search engine, we can get the basic situation of overlap between the results. Some results are retrieved by all four search engines, while most of the results are only returned by one search engine. In Table 1, we report the overview of the search results in three cases: top 10, top 20 and top 50. It gives the number of results retrieved, on average, out of 58 queries. From this table, we can see that over 75 percent of the total distinct results are retrieved by only one search engine, and only about 3 percent of the total distinct results are retrieved by all four search engines. Table 2 shows the case where the maximum and minimum distinct search results are retrieved. For instance, considering the top 50 results from each search engine, the maximum number of distinct returned results is 185 when submitting the query "xxx". At the same time, the queries which achieved maximum and minimum overlap in all four search engines are displayed in Table 3. For example, considering the top 20 results from each search engine, there are seven results retrieved by all four search engines when submitting the query "google".
Case | Top 10 | Top 20 | Top 50 |
---|---|---|---|
The number of results returned by only 1 search engine | 22.36 (77.32%) | 44.90 (77.29%) | 107.91 (75.49%) |
The number of results returned by 2 search engines | 3.84 (13.28%) | 7.24 (12.46%) | 20.64 (14.44%) |
The number of results returned by all 3 search engines | 1.79 (6.19%) | 4.14 (7.13%) | 9.84 (6.88%) |
The number of results returned by all 4 search engines | 0.93 (3.21%) | 1.81 (3.12%) | 4.55 (3.19%) |
The number of total distinct results | 28.92 | 58.09 | 142.94 |
Case | Top 10 | Top 20 | Top 50 |
---|---|---|---|
Maximum (The Query) | 39 (free porn) | 75 (porn) | 185 (xxx) |
Minimum (The Query) | 17 (google) | 38 (google) | 106 (baby names ) |
Average | 28.92 | 58.09 | 142.94 |
Case | Top 10 | Top 20 | Top 50 |
---|---|---|---|
Maximum (The Query) | 4 (clip art) | 7 (google) | 15 (clip art) |
Minimum |
|
|
|
Average | 0.93 | 1.81 | 4.55 |
As we noted in the last section, we divide all 58 queries into three categories. The queries in Category A are mainly composed of a specific name of a web site, a company or a product. Usually there exists a web site for this kind of query. In Category B, the queries are very general terms, such as map, hotels, jokes , etc. In Category C, the queries are mainly related to adult content.
In this experiment, we compared the search results for different categories. At first we compare the top1 hit from each search engine. As illustrated in Table 4, the search engines tend to agree with each other on Category A, but they behave totally differently on Category C. For each query in Category A, any two search engines will return same hit (web site) at the top1 position in 85 percent of the cases, while only a 6.25 percent overlap occurs for Category C.
Category | A(specific) | B(General) | C(adult) |
---|---|---|---|
Number of Queries | 10 | 40 | 8 |
(Google-AltaVista) | 9 (90%) | 12 (30%) |
|
(Google-AlltheWeb) | 8 (80%) | 13 (32.5%) | 1 (12.5%) |
(Google-WiseNut) | 8 (80%) | 15 (37.5%) | 2 (25%) |
(AltaVista-AlltheWeb) | 9 (90%) | 6 (15%) |
|
(AltaVista-Wisenut) | 9 (90%) | 7 (17.5%) |
|
(AlltheWeb-WiseNut) | 8 (80%) | 13 (32.5%) |
|
Average | 8.5 (85%) | 11 (27.5%) | 0.5 (6.25%) |
In Table 5, we present the maximum, minimum, and average number of totally distinct search results for the queries in each category. In all three cases (top 10, top 20, top 50), we obtain the most results for the queries in Category C, and the fewest results for queries in Category A. But, the difference between the queries in Category A and Category B decreases when we fetch more results for merging from each search engine.
Case | Category | Maximum | Minimum | Average |
---|---|---|---|---|
Top 10 | A (specific) | 17 | 33 | 24.4 |
Top 10 | B (general) | 19 | 36 | 28.8 |
Top 10 | C (adult) | 31 | 39 | 35.25 |
Top 20 | A (specific) | 38 | 65 | 52.9 |
Top 20 | B (general) | 44 | 73 | 56.78 |
Top 20 | C (adult) | 64 | 75 | 71.13 |
Top 50 | A (specific) | 108 | 157 | 136.7 |
Top 50 | B (general) | 106 | 167 | 138.4 |
Top 50 | C (adult) | 160 | 185 | 173.5 |
In Tables 6, 7 and 8, we present the average overlap of the top N (N=10, 20 or 50, respectively) search results, between any two search engines, out of all 58 queries in three categories. We can see that AltaVista and WiseNut have the lowest overlap on the queries in Category A; Google and WiseNut have the highest overlap over the queries in Category B and Category C.
Category | A(specific) | B(General) | C(adult) |
---|---|---|---|
Overlap(Google-AltaVista) | 3.6 | 2.13 | 1.13 |
Overlap(Google-AlltheWeb) | 3.4 | 3.5 | 0.75 |
Overlap(Google-WiseNut) | 3 | 3.9 | 2 |
Overlap(AltaVista-AlltheWeb) | 3.3 | 1.43 | 0.5 |
Overlap(Altavista-WiseNut) | 2.6 | 1.73 | 0.62 |
Overlap(AlltheWeb-WiseNut) | 2.6 | 3.08 | 0.5 |
Average | 3.08 | 2.63 | 0.917 |
Category | A(specific) | B(General) | C(adult) |
---|---|---|---|
Overlap(Google-AltaVista) | 5.8 | 4.72 | 2.63 |
Overlap(Google-AlltheWeb) | 6.6 | 7.55 | 1.25 |
Overlap(Google-WiseNut) | 6.3 | 8.15 | 3.62 |
Overlap(AltaVista-AlltheWeb) | 5.2 | 3.6 | 0.63 |
Overlap(AltaVista-WiseNut) | 4.5 | 3.55 | 1.5 |
Overlap(AlltheWeb-WiseNut) | 5.9 | 6.0 | 0.86 |
Average | 5.72 | 5.60 | 1.75 |
Category | A(specific) | B(General) | C(adult) |
---|---|---|---|
Overlap(Google-AltaVista) | 14.6 | 13.93 | 7.38 |
Overlap(Google-AlltheWeb) | 15.2 | 18.8 | 3.5 |
Overlap(Google-WiseNut) | 13.9 | 19.68 | 9.75 |
Overlap(AltaVista-AlltheWeb) | 11.8 | 10.28 | 1.5 |
Overlap(AltaVista-WiseNut) | 9.5 | 10.2 | 4.38 |
Overlap(AlltheWeb-WiseNut) | 12.2 | 14.48 | 2 |
Average | 12.87 | 14.56 | 4.75 |
Table 9 shows the average overlap of the top N (N=10, 20 or 50) search results, between one search engine and any other search engine, out of all 58 queries in three categories. For instance: Overlap(Google)=1/3(Overlap(Google-AltaVista)+Overlap(Google-AlltheWeb)+ Overlap(Google-WiseNut)). The high value of average overlap means that the search engine is highly supported by all other search engines. With respect to queries in Category B, the order of the search engines from high to low, according to average overlap, is Google, WiseNut, AlltheWeb and AltaVista, for all three cases (top10, top 20, top50). Different from Category B, the order for the queries in Category C for all three cases is Google, WiseNut, AltaVista and AlltheWeb.
Case | Category | | AltaVista | AlltheWeb | WiseNut |
---|---|---|---|---|---|
Top10 | A | 3.33 | 3.17 | 3.1 | 2.73 |
Top10 | B | 3.18 | 1.76 | 2.67 | 2.9 |
Top10 | C | 1.29 | 0.75 | 0.58 | 1.04 |
Top20 | A | 6.23 | 5.17 | 5.9 | 5.57 |
Top20 | B | 6.81 | 3.96 | 5.72 | 5.9 |
Top20 | C | 2.5 | 1.58 | 0.92 | 2 |
Top50 | A | 14.57 | 11.97 | 13.07 | 11.87 |
Top50 | B | 17.47 | 11.47 | 14.52 | 14.78 |
Top50 | C | 6.88 | 4.42 | 2.33 | 5.38 |
However, in Category A, we don't have a consistent order for the queries for all three cases (top 10, top 20, top 50). In Category A, Google always obtains the highest overlap, and the difference among all four search engines is very small. As shown in Table 9, Google always achieves the highest overlap among all cases for the queries in all three categories.
In the previous section, we presented a method ( Kendall tau distance ) for computing the distance between two rank lists retrieved by different search engines. Since low overlap makes the distance less meaningful, we will eliminate some low overlap cases and consider the following cases for the experiment: Category A, top 20; Category A, top 50; Category B, top 20; Category B, top 50.
In this experiment, we calculated the Kendall tau distance between two search results retrieved by different search engines. Table 10 presents the results of the mean distance over any two search engines out of all queries in Categories A and B in two cases (top 20 and top 50). Since the overlap is very small for the queries in Category C, we don't calculate the distance for the queries in Category C. We can see that there are no obvious differences among all search engine pairs. Since all search engines tend to agree with the top 1 for queries in Category A, the distance for Category A is obviously smaller than the one in Category B.
Category | A(Speicfic) | B(general) | A(specific) | B(general) |
---|---|---|---|---|
Top N | 20 | 20 | 50 | 50 |
Google-AltaVista | 0.215 | 0.404 | 0.282 | 0.43 |
Google-AlltheWeb | 0.244 | 0.365 | 0.33 | 0.366 |
Google-WiseNut | 0.242 | 0.326 | 0.294 | 0.348 |
AltaVista-AlltheWeb | 0.176 | 0.424 | 0.333 | 0.402 |
AltaVista-WiseNut | 0.152 | 0.369 | 0.284 | 0.432 |
AlltheWeb -WiseNut | 0.243 | 0.385 | 0.334 | 0.381 |
In Table 11, we present the average Kendall tau distance between each search engine and any other search engines. We can see that AltaVista has the largest distance for the queries in Category B, while it has the smallest distance for the queries in Category A. In Category B, Google has the smallest distance.
Category | Top N | | AltaVista | AlltheWeb | WiseNut |
---|---|---|---|---|---|
A | 20 | 0.243 | 0.178 | 0.229 | 0.197 |
A | 50 | 0.302 | 0.3 | 0.332 | 0.304 |
B | 20 | 0.358 | 0.397 | 0.384 | 0.356 |
B | 50 | 0.381 | 0.423 | 0.382 | 0.386 |