Performance of other Video Quality Measures

As mentioned before in Section 3.2 the most known measures to evaluate video quality are the ITS and EPFL metrics. The correlation coefficient obtained when testing it for some video scenes (see Figure 6.6) without network impairments is about 94% as reported in [143]. However, when this metric is used to evaluate the quality when varying only the bit rate, the performance becomes very bad namely for low bit rate. This can be seen from Figure 6.9 and Figure 6.13. In addition, this measure was designed basically to evaluate the quality of video impaired by encoding distortions. It is not able to evaluate the quality when varying the most basic encoding variable, namely the bit rate. Furthermore, as other models, it requires the access to both the original and the processed signals. This makes it not suitable to real-time application. More details about this measure is given in [138,142,151]. The other ones are the EPFL metrics, namely, the Moving Pictures Quality Metric (MPQM), the Color MPQM (CMPQM), and the Normalization Video Fidelity Metric (NVFM) [135]. All these metrics are designed for high quality video broadcasting evaluation as stated in [135, p. 90] it is concluded that ``Eventually, it is to be noted that such metrics operate best on high quality video sequences and are thus best suited for quality assessment of broadcasted sequences rather than very low bit rates scenes. This is due to the fact that the vision model is essentially a threshold model and thus operates better when the stimulus to deal with, the distortion in this case, is close to threshold, i.e. for high quality scenes.'' From the obtained results we can see from Figure 6.7 as well as the next Figures that this is not always true. As shown, the metric cannot evaluate the quality in the presence of loss. Logically, it is expected that the quality becomes better when the bit rate increases (reducing the encoding artifacts). But once, a very small amount of loss is applied, the metric output does the contrary, the quality decreases with the increase of the bite rate.

Figure 6.6: The performance of the ITS metric to evaluate video quality. This Figure is taken from [143, p. 508].

  

Figure 6.7: Quality assessment by MPQM as a function of the bitrate and the loss rate. This Figure is taken from [135, p. 88].

Another drawback for these metrics is that the computational complexity is too high, and so it may limit their use on real-time multimedia evaluation in the Internet. This is conformed by what is quoted from [135, p. 88] ``Due to computational complexity and memory management, the MPQM could only be applied to 32 frames of the sequences. It has been chosen to always use the first 32 frames of the video stream.''. Note that for a video sequence encoded at 30 frames/sec., there are 300 frames in only 10 sec. In addition, we quoted from [135, p. 135] ``...From an implementation point of view, some efforts have to be put in reducing the computational load of such vision models. Powerful architectures are still needed to obtain fast results with the models [128]...'' What is referred to by ``Powerful architectures'' are supercomputing ones as the citation 128 indicates. Note that MPQM metric is simpler and less accurate than CMPQM metric and that 32 frames correspond to 1 sec. of video. Similar to ITS metric, MPQM and CMPQM cannot operate well for low bit rates, this is shown in Figures 6.8, 6.10, 6.12 and 6.13. As we can see, MPQM and CMPQM give a score of 3 instead of 1 on the 5-point quality scale (as it should be at very low bit rate). NVFM gives better results for some sequences, but fails for others, as shown in Figure 6.11 it gives a score of 2 instead of 1 for the ``Mobile & Calander'' video sequence. Regarding the correlation with subjective quality tests, it is not very good as we can see from Figure 6.13, the vertical solid lines. Unfortunately, there are no available numerical values for the correlation coefficient with subjective measures, but we can see it is not expected to be good.

Figure 6.8: MPQM quality assessment of MPEG-2 video as a function of the bit rate. This Figure is taken from [135, p. 82].

  

Figure 6.9: The quality assessment by the ITS model of MPEG-2 video as a function of the bit rate. This Figure is taken from [135, p. 83].

Figure 6.10: CMPQM quality assessment of MPEG-2 video as a function of the bit rate. This Figure is taken from [135, p. 83].

  

Figure 6.11: NVFM quality assessment of MPEG-2 video as a function of the bit rate. This Figure is taken from [135, p. 83].

Figure 6.12: Comparison of the subjective data (vertical solid lines) against MPQM, CMPQM and NVFM metrics for the video sequence ``Mobile & Calendar''. This Figure is taken from [135, p. 84].

  

Figure 6.13: Comparison of the subjective data (vertical solid lines) against MPQM, CMPQM, NVFM and ITS metrics for the video sequence ``Basket Ball''. This Figure is taken from [135, p. 84].