EXPERIMENT RESULTS


We carried out the experiment on August 2, 2002. All 58 queries were submitted to Google, AltaVista, AlltheWeb and WiseNut. The top 50 results from each search engine were retrieved for further analysis according to the principles described in last section. Here, we give our main results of the experiment.

Overview

After analyzing the results returned by each search engine, we can get the basic situation of overlap between the results. Some results are retrieved by all four search engines, while most of the results are only returned by one search engine. In Table 1, we report the overview of the search results in three cases: top 10, top 20 and top 50. It gives the number of results retrieved, on average, out of 58 queries. From this table, we can see that over 75 percent of the total distinct results are retrieved by only one search engine, and only about 3 percent of the total distinct results are retrieved by all four search engines. Table 2 shows the case where the maximum and minimum distinct search results are retrieved. For instance, considering the top 50 results from each search engine, the maximum number of distinct returned results is 185 when submitting the query "xxx". At the same time, the queries which achieved maximum and minimum overlap in all four search engines are displayed in Table 3. For example, considering the top 20 results from each search engine, there are seven results retrieved by all four search engines when submitting the query "google".

Table 1: Number of Results Retrieved on Average Out of All 58 Queries in Three Cases: Top 10, Top 20 and Top 50

Case

Top 10

Top 20

Top 50

The number of results returned by only 1 search engine

22.36 (77.32%)

44.90 (77.29%)

107.91 (75.49%)

The number of results returned by 2 search engines

3.84 (13.28%)

7.24 (12.46%)

20.64 (14.44%)

The number of results returned by all 3 search engines

1.79 (6.19%)

4.14 (7.13%)

9.84 (6.88%)

The number of results returned by all 4 search engines

0.93 (3.21%)

1.81 (3.12%)

4.55 (3.19%)

The number of total distinct results

28.92

58.09

142.94

Table 2: Maximum, Minimum, and Average Number of Distinct Results Retrieved Out of All 58 Queries in Three Cases: Top 10, Top 20 and Top 50

Case

Top 10

Top 20

Top 50

Maximum (The Query)

39 (free porn)

75 (porn)

185 (xxx)

Minimum (The Query)

17 (google)

38 (google)

106 (baby names )

Average

28.92

58.09

142.94

Table 3: Maximum, Minimum, and Average Number of Distinct Results Retrieved by All Four Search Engines

Case

Top 10

Top 20

Top 50

Maximum (The Query)

4 (clip art)

7 (google)

15 (clip art)

Minimum

Average

0.93

1.81

4.55

Difference Among Categories

As we noted in the last section, we divide all 58 queries into three categories. The queries in Category A are mainly composed of a specific name of a web site, a company or a product. Usually there exists a web site for this kind of query. In Category B, the queries are very general terms, such as map, hotels, jokes , etc. In Category C, the queries are mainly related to adult content.

In this experiment, we compared the search results for different categories. At first we compare the top1 hit from each search engine. As illustrated in Table 4, the search engines tend to agree with each other on Category A, but they behave totally differently on Category C. For each query in Category A, any two search engines will return same hit (web site) at the top1 position in 85 percent of the cases, while only a 6.25 percent overlap occurs for Category C.

Table 4: Number of Queries Having Same Top 1 (First) Result Between Search Engine Pairs Out of 58 Queries in Three Categories

Category

A(specific)

B(General)

C(adult)

Number of Queries

10

40

8

(Google-AltaVista)

9 (90%)

12 (30%)

(Google-AlltheWeb)

8 (80%)

13 (32.5%)

1 (12.5%)

(Google-WiseNut)

8 (80%)

15 (37.5%)

2 (25%)

(AltaVista-AlltheWeb)

9 (90%)

6 (15%)

(AltaVista-Wisenut)

9 (90%)

7 (17.5%)

(AlltheWeb-WiseNut)

8 (80%)

13 (32.5%)

Average

8.5 (85%)

11 (27.5%)

0.5 (6.25%)

In Table 5, we present the maximum, minimum, and average number of totally distinct search results for the queries in each category. In all three cases (top 10, top 20, top 50), we obtain the most results for the queries in Category C, and the fewest results for queries in Category A. But, the difference between the queries in Category A and Category B decreases when we fetch more results for merging from each search engine.

Table 5: Number of Distinct Results Retrieved Out of 58 Queries in Three Categories for Three Cases: Top 10, 20 and 50

Case

Category

Maximum

Minimum

Average

Top 10

A (specific)

17

33

24.4

Top 10

B (general)

19

36

28.8

Top 10

C (adult)

31

39

35.25

Top 20

A (specific)

38

65

52.9

Top 20

B (general)

44

73

56.78

Top 20

C (adult)

64

75

71.13

Top 50

A (specific)

108

157

136.7

Top 50

B (general)

106

167

138.4

Top 50

C (adult)

160

185

173.5

Overlap of the Search Results

In Tables 6, 7 and 8, we present the average overlap of the top N (N=10, 20 or 50, respectively) search results, between any two search engines, out of all 58 queries in three categories. We can see that AltaVista and WiseNut have the lowest overlap on the queries in Category A; Google and WiseNut have the highest overlap over the queries in Category B and Category C.

Table 6: Average Overlap of Top10 Results Between Any Two Search Engines Out of 58 Queries in Three Categories

Category

A(specific)

B(General)

C(adult)

Overlap(Google-AltaVista)

3.6

2.13

1.13

Overlap(Google-AlltheWeb)

3.4

3.5

0.75

Overlap(Google-WiseNut)

3

3.9

2

Overlap(AltaVista-AlltheWeb)

3.3

1.43

0.5

Overlap(Altavista-WiseNut)

2.6

1.73

0.62

Overlap(AlltheWeb-WiseNut)

2.6

3.08

0.5

Average

3.08

2.63

0.917

Table 7: Average Overlap of Top 20 Results Between Any Two Search Engines Out of 58 Queries in Three Categories

Category

A(specific)

B(General)

C(adult)

Overlap(Google-AltaVista)

5.8

4.72

2.63

Overlap(Google-AlltheWeb)

6.6

7.55

1.25

Overlap(Google-WiseNut)

6.3

8.15

3.62

Overlap(AltaVista-AlltheWeb)

5.2

3.6

0.63

Overlap(AltaVista-WiseNut)

4.5

3.55

1.5

Overlap(AlltheWeb-WiseNut)

5.9

6.0

0.86

Average

5.72

5.60

1.75

Table 8: Average Overlap of Top 50 Results Between Any Two Search Engines Out of 58 Queries in Three Categories

Category

A(specific)

B(General)

C(adult)

Overlap(Google-AltaVista)

14.6

13.93

7.38

Overlap(Google-AlltheWeb)

15.2

18.8

3.5

Overlap(Google-WiseNut)

13.9

19.68

9.75

Overlap(AltaVista-AlltheWeb)

11.8

10.28

1.5

Overlap(AltaVista-WiseNut)

9.5

10.2

4.38

Overlap(AlltheWeb-WiseNut)

12.2

14.48

2

Average

12.87

14.56

4.75

Table 9 shows the average overlap of the top N (N=10, 20 or 50) search results, between one search engine and any other search engine, out of all 58 queries in three categories. For instance: Overlap(Google)=1/3(Overlap(Google-AltaVista)+Overlap(Google-AlltheWeb)+ Overlap(Google-WiseNut)). The high value of average overlap means that the search engine is highly supported by all other search engines. With respect to queries in Category B, the order of the search engines from high to low, according to average overlap, is Google, WiseNut, AlltheWeb and AltaVista, for all three cases (top10, top 20, top50). Different from Category B, the order for the queries in Category C for all three cases is Google, WiseNut, AltaVista and AlltheWeb.

Table 9: Average Overlap of the TopN (N=10,20 or 50) Results Between Each Search Engine and Any Other Search Engines Out of All 58 Queries in Three Categories

Case

Category

Google

AltaVista

AlltheWeb

WiseNut

Top10

A

3.33

3.17

3.1

2.73

Top10

B

3.18

1.76

2.67

2.9

Top10

C

1.29

0.75

0.58

1.04

Top20

A

6.23

5.17

5.9

5.57

Top20

B

6.81

3.96

5.72

5.9

Top20

C

2.5

1.58

0.92

2

Top50

A

14.57

11.97

13.07

11.87

Top50

B

17.47

11.47

14.52

14.78

Top50

C

6.88

4.42

2.33

5.38

However, in Category A, we don't have a consistent order for the queries for all three cases (top 10, top 20, top 50). In Category A, Google always obtains the highest overlap, and the difference among all four search engines is very small. As shown in Table 9, Google always achieves the highest overlap among all cases for the queries in all three categories.

Distance of the Search Results

In the previous section, we presented a method ( Kendall tau distance ) for computing the distance between two rank lists retrieved by different search engines. Since low overlap makes the distance less meaningful, we will eliminate some low overlap cases and consider the following cases for the experiment: Category A, top 20; Category A, top 50; Category B, top 20; Category B, top 50.

In this experiment, we calculated the Kendall tau distance between two search results retrieved by different search engines. Table 10 presents the results of the mean distance over any two search engines out of all queries in Categories A and B in two cases (top 20 and top 50). Since the overlap is very small for the queries in Category C, we don't calculate the distance for the queries in Category C. We can see that there are no obvious differences among all search engine pairs. Since all search engines tend to agree with the top 1 for queries in Category A, the distance for Category A is obviously smaller than the one in Category B.

Table 10: Average Kendall Distance of the Top N (N=20 or 50) Results Between Any Two Search Engines Out of All Queries in Category A and B

Category

A(Speicfic)

B(general)

A(specific)

B(general)

Top N

20

20

50

50

Google-AltaVista

0.215

0.404

0.282

0.43

Google-AlltheWeb

0.244

0.365

0.33

0.366

Google-WiseNut

0.242

0.326

0.294

0.348

AltaVista-AlltheWeb

0.176

0.424

0.333

0.402

AltaVista-WiseNut

0.152

0.369

0.284

0.432

AlltheWeb -WiseNut

0.243

0.385

0.334

0.381

In Table 11, we present the average Kendall tau distance between each search engine and any other search engines. We can see that AltaVista has the largest distance for the queries in Category B, while it has the smallest distance for the queries in Category A. In Category B, Google has the smallest distance.

Table 11: Average Kendall Distance of the Top N (N=20 or 50) Results Between Each Search Engine and Any Other Search Engines Out of All 58 Queries in Category A and Category B

Category

Top N

Google

AltaVista

AlltheWeb

WiseNut

A

20

0.243

0.178

0.229

0.197

A

50

0.302

0.3

0.332

0.304

B

20

0.358

0.397

0.384

0.356

B

50

0.381

0.423

0.382

0.386




(ed.) Intelligent Agents for Data Mining and Information Retrieval
(ed.) Intelligent Agents for Data Mining and Information Retrieval
ISBN: N/A
EAN: N/A
Year: 2004
Pages: 171

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net