Proposed Validation Process for IPTV Perceptual Quality Measurement (PQM) ATIS IIF QoSM Committee Sept. 2008
On-demand Validation • PQM standards are useful, but they • Do not encourage continuous improvement • Become outdated • Proposal: On-demand validation process • Individual model testing • Quick turn-around times • Modifying & re-testing models possible • Secret, re-usable database • “Certification” of models
Scope • Define test process for on-demand testing • Define test plan(s) • Define reporting process • Parties involved: • Model Developers • Independent Test Lab(s) • Model Users (e.g. service providers)
Content Library • Secret SRC and PVS • Documentation of SRC & HRC available publicly (thumbnails, text description) • Maintained by independent labs • Avoids tuning possibilities • Re-usable, i.e. does not become “known content” after test • May grow over time
Content Library • Cover different genres, applications and service scenarios • Linear TV, VoD, mobile TV, etc. • SD/HD • Model developers can choose validation against specific scenarios
Certification vs. Standardization • Certification for different categories(e.g. HD video-only PQM) • Specify performance thresholds(may be scenario-dependent) • Define certification classes/levels (e.g. Class A for models above 90% corr.) • Encourages competition • Allows vendors to price/market based on certification level
Results Reporting • Detailed report • For model developer’s use • Summary report • Part of certification (i.e. becomes public once developer releases certification results) • Report prediction performance as well as computational complexity (runtime)
Discussion • Comments, feedback, suggestions? • Who would like to contribute? • Who can validate/certify? • Is this a process model VQEG would consider adopting and/or executing?