310 likes | 461 Vues
Results of the ATIS/T1A1.1 Ad Hoc Group on Full-Reference Video Quality Metrics (FR-VQM) VSF Meeting October 3, 2001 John Pearson Sarnoff Corporation jpearson@sarnoff.com. Take Home Messages. Tariff’s can now include Visual Quality Metrics (Full Reference)
E N D
Results of the ATIS/T1A1.1Ad Hoc Group on Full-Reference Video Quality Metrics (FR-VQM)VSF MeetingOctober 3, 2001John PearsonSarnoff Corporationjpearson@sarnoff.com
Take Home Messages • Tariff’s can now include Visual Quality Metrics (Full Reference) • The basis for this is a family of 4 Technical Reports by ATIS/T1A1 • The T1A1 approach is extensible to additional Visual Quality Metrics, and does NOT establish a Standard
Outline • Why is measuring Visual Quality important? • Why is measuring Visual Quality hard? • International Standards for VQM’s • T1A1 Technical Reports
FR-VQM Needs of US Telecom Q-?? Q-A Q-B • Digital video processing can create objectionable noise • End-to-End QoS across the networks of multiple companies requires agreement on Quality at Transfer Points (Tariffs) • Tariff’s require ANSI sanctioned technical documentation Company Auses VQM-A Company Buses VQM-B Site of Video Origination(e.g., Denver) Transfer Between Network A & B Site of Video Consumption(e.g., Mexico City)
Digital Video Creates “Patterned” Noise ... Human visual response to patterned noise highly non-linear ... Random “Analog” Noise Blocky “Digital” Noise MSE = 27.10 MSE = 21.26 Measures like MSE suitable for Analog noise no longer work for Digital noise
Patterned noise in the sky much more perceptible even though much smaller in terms of pixel differences Source Frame Difference Map Codec Frame
Visual Quality Metrics... correlate well across scene types, unlike MSE ... Visual Discrimination Model Mean-Squared Error Mean of 80 trials for 20 subjects Bars show 5% confidence intervals
Vital Role of Subjective Database • Goal of VQMs is to approximate subjective quality assessments (SQA) • The relevance of the SQA depends on: • Test sequences (SRC’s) • Distortion generators (HRC’s) • Viewing conditions and testing protocols • Producing a relevant SQA is hard
Three Kinds of VQM’s • Full Reference (FR) • a double-ended method and is the subject of this Technical Report. • Reduced Reference (RR) • only reduced video reference information is available. This is also a double-ended method. • No Reference (NR) • no reference video signal or information is available. This is a single-ended method. • It is generally believed that the FR method will provide the most accurate measurement results while the RR and NR methods will be more convenient for QoS monitoring. • The T1A1 Technical Reports concern FR methods
International Standards Progress • VQEG may be several years from recommending a FR-VQM standard to ITU • Its possible that no single FR-VQM will be a clear “winner” • The FR-VQM field is young, and significant, steady improvements are expected over the next decade • It’s possible that several different FR-VQMs may gain industry acceptance
T1A1.1 FR-VQM Strategy… an extensible family of TR’s for FR-VQ, enabling Industry to move ahead without Standards ... • Provide guidelines for how Industry can • specify its specific FR-VQM needs • assess the suitability of existing documented FR-VQMs • drive the development by FR-VQM proponents of new/improved FR-VQM algorithms and products • inter-operate with different FR-VQMs • Provide guidelines for how FR-VQMs can be • documented in algorithms, accuracy and limitations • quantitatively cross-calibrated to each another • Extensible framework enabling addition of FR-VQMs • Start by specifying two already disclosed FR-VQMs • Stimulates continued FR-VQM innovation
Family of Technical Reports • TR A1: Accuracy and Cross-Calibration (Mike Brill, Sarnoff) • defines accuracy (statistical analysis), limitations of a FR-VQM • defines transformation to common scale, for cross-calibration with other applicable FR-VQMs • TR A2: Normalization Methods (David Fibush, Tektronix) • applied to source and processed video before VQM calculation • e.g., spatial/temporal registration, gain/level offset calibration, ... • may utilize special test signals • TR A3: Peak Signal to Noise Ratio (Steve Wolf, NTIA) • Specify PSNR VQM, following TR A1 and TR A2 guidelines • TR A4: Objective Perceptual FR-VQM Using a JND-Based Full Reference Technique (David Fibush, Mike Brill) • Specify JND-based FR-VQM, following TR A1 and TR A2 guidelines
TRA1 Defines Basic Methods: LIMITATIONS • “How to” specify VQM accuracy • with respect to subjective assessments • based on defined statistical analysis • “How to” specify VQM scope/limitations • type of scene content (“signal”) • high/low motion, color/b&w, interlaced/progressive • type/severity of artifacts (“noise”) • e.g., encoding techniques, bit-rates, blurring, blockiness • subjective testing characteristics • behavior with viewing distance, resolution, gamma, … • expert vs non-expert viewers • “How to” cross-calibrate VQMs • determination of mathematical transformation relating one VQM’s outputs to another’s SCOPE Works well, & has been well tested here
Normalization Requirements JND/PQR PSNR
Logistic-mapped PQR for Common Scale… provides approach for cross-calibration...
Accuracy -- 3 Methods • RMSE • Resolving Power • Classification of Errors
RMSE • RMSE: root mean square error between subjective and objective normalized scores
Progress • T1A1.1 Ad Hoc Group created Feb. 2001, co-chairs John Grigg, John Pearson • Mail Ballot Approval August 2001 • Approved by T1A1.1 25 September 2001 • Approved at Plenary meeting of T1A1, 28 September 2001