6. Accelerating Relevance Feedback

While the process of relevance feedback generally kicks in once the user has seen some results, it is desirable to provide the benefits of relevance feedback from the very beginning of the user's search process. A basic limitation of relevance feedback is that it is generally not available for the initial query since it needs the results and user judgements to act. Indeed, all the techniques discussed so far suffer from this limitation.

It is possible to enhance the initial user query with the information gained from relevance feedback in queries made by other users. The field of Collaborative Filtering [10] has a rich literature based on the premise that a textual document should be returned to a query not based on its content, but on a measure of popularity based on how many users selected it based on their queries. More broad than collaborative filtering are recommender systems that exploit the information acquired and created by one user in her interaction with the system to assist other users [22].

FeedbackBypass [4] is a prominent example of applying collaborative filtering techniques to multimedia retrieval. The objective is to sidestep many feedback iterations and directly go to an optimal query representation given the initial user specification. It focuses on a single feature space approach as presented in section 4. It represents queries as point p in a multidimensional space, together with a distance function d for the feature, that is, the pair <p,d> specifies a query. FeedbackBypass supports the weighted Euclidean and Mahalanobis distance functions as described in section 4, that is, we can represent the distance function d as a set of weights w for either distance function. Since we can capture d with its corresponding set of weights w, we can represent the query as <p,w>. There are two steps in its operation:

Build a mapping from initial to optimal query specifications. The objective is to build the mapping:
- p⁰ → (p^opt, w^opt)
A user starts with an initial query point p⁰. She provides relevance feedback and eventually arrives at an optimal query representation: <p^opt, w^opt>. This end result of several feedback iterations is encoded as an optimal offset Δp=p^opt - p⁰ and the weights w^opt, that is, the difference between the initial and final query points and the optimal set of weights. The initial query point p⁰, is used as the key into a multidimensional data structure with (Δp, w^opt) as the values. To avoid storing all possible mappings, FeedbackBypass uses a simplex tree to merge "close" initial points together. Over time, this mapping is populated with the results of the collective feedback of many users.
Given a mapping p⁰ → (p^opt, w^opt) and a user's initial query point, find the corresponding optimal query specification. To circumvent many feedback iterations, the initial query point p⁰ is used as a key into the multidimensional index and the corresponding pair (Δp, w^opt) is retrieved. An optimal query representation is computed as <p⁰ +Δp, w^opt) and used to execute the user's query.

In essence, FeedbackBypass keeps a history of how query point movement and re-weighting evolved and then applies the learned information to new queries. Over time, as users continue to give relevance feedback, the system keeps learning its mapping, and therefore improving its results.

Another system that uses collaborative filtering for browsing images in the context of a museum web site is described in [14]. It combines multiple visual features (content based filtering) and user recommendations (collaborative filtering) to dynamically guide the user in a virtual museum. This approach predicts the interest rating of a user in an image by matching the visual features of the image to other images and in turn using those images' recommendation rating to form a final estimate. The collaborative component produces a distance value ρ∈[0,1] based on Pearson correlation [14]. The content-based component produces distances for two features, color histogram and texture. The final distance for an image is computed as a weighted summation of the results from visual features and the correlation coefficient:

The weights μⁱ are set experimentally.

The overall results for collaborative filtering approaches indicate that the results of a retrieval system improve substantially when taking into consideration the aggregated feedback from all users over the lifetime of the system.