5.1 Evaluation of Results

The effectiveness of the algorithm was evaluated on the whole MPEG-7 test data set using the precision and recall measures, which are widely used in the field of information retrieval.

(8.24)

(8.25)

The twin-comparison algorithm, which is the most commonly used technique today to detect GTs, was first run on the data set. Here, the twin-comparison algorithm was implemented using the 64-bin color histogram, but without performing motion detection. It can be noticed in Table 8.4 that the twin-comparison method has a very low recall and precision when detecting GTs. This is because the algorithm is threshold-sensitive, and requires different thresholds for different video types

Table 8.4: Result Using Twin-Comparison
Transition Type	Precision	Recall
CUT	40%	92%
GT	15%	58%

Next, we apply the TMRA algorithm using the same feature space (64-bin color histogram). Table 8.5 summarizes the initial result that demonstrates that TMRA approach outperforms the twin-comparison method especially on GT detection. The TMRA algorithm could achieve very high recall of over 97% for both CUT's and GTs. However, there is too much false detection that made our precision not satisfactory. To demonstrate that the TMRA framework is independent of the content feature used, the framework has also been applied in other feature spaces like DCHistogram and DC64 in the compressed domain. The results are respectively shown in Table 8.6 and Table 8.7. The result in Table 8.6 shows that the TMRA can achieve good results in this feature space. It can also be seen that the precision for GT detection is increased. This is because DC histogram smoothes the noise more effectively. Thus, less false detection was introduced in the compressed domain.

Table 8.5: Result on TMRA using 64-bin Histogram Without Elimination Phase
Transition Type	Precision	Recall
CUT	68%	98%
GT	51%	97%

Table 8.6: Result on TMRA using DC Histogram Without Elimination Phase
Transition Type	Precision	Recall
CUT	75%	97%
GT	68%	97%

Table 8.7: Result on TMRA using DC64 Without Elimination Phase
Transition Type	Precision	Recall
CUT	66%	98%
GT	53%	96%

Table 8.6 and Table 8.7 shows that the use of compressed domain feature 64-bin DC histogram and DC64 is effective. This resulted in the improvement in performance over the use of 64-bin color histogram (uncompressed domain). This is extremely encouraging as it demonstrates the effectiveness of using compressed domain features in video boundary detection. Among the two-feature space for compressed video, the 64-bin DC histogram gives the best performance. In particular, it achieves high recall of 97% and precision of over 68% for GT's. The low precision of only 68% for GTs is due to the high number of false detections. Most of the false GT detections were due to motion (either camera motion or object motion). Thus, it is necessary to detect motions and to distinguish phenomena of motion change from transitions.

After applying the elimination phase using the motion-based feature MA64, many false transitions were eliminated and resulted in very high precision with little sacrifice in recall values. From the results shown in Table 8.8, it is observed that the use of motion information can increase the precision from 68% to 87% for GT's while maintaining excellent recall value of 90%.

Table 8.8: Result on TMRA Integrated with Elimination Phase Using DC64 and MA64
Transition Type	Precision	Recall
CUT	96%	92%
GT	87%	90%

All the results show that the TMRA framework is effective for video transition detections and can be applied with different frame features in raw domain or in compressed domain.