Recent improvements in transmission quality assessment : Background noise transmission

1. Recent improvements in transmission quality assessment : Background noise transmission Results of STF 273, STF 294 Jean-Yves Monfort, France Telecom ETSI TC STQ Chairman ETSI STQ, Taiwan Workshop, February 13th, 2006

2. STFs • STF : Special Task Force • Founded by ETSI (STF 273) and e-Europe (STF 294) • Several Experts involved in a project  new inputs in standards • STF 273 Speech quality performance in the presence of background noise  A noise data base, freely available  EG 202 396-1 • STF 294 Improving the quality of eEurope wideband speech applications by developing a standardised performance testing and evaluation methodology for background noise transmission  EG 202 396-2 ETSI STQ, Taiwan Workshop, February 13th, 2006

3. STF 273 Speech quality performance in the presence of background noise Objectives : a close-to-original recording and reproduction system The background noise recording technique should be– Easy to use – Easy to calibrate – Capable of wideband recording– Available at reasonable costs– Mostly compatible with existing standards and procedures used in telecommunications testing ETSI STQ, Taiwan Workshop, February 13th, 2006

4. STF 273 Speech quality performance in the presence of background noise Objectives : a close-to-original recording and reproduction system The background noise simulation (reproduction) arrangement should be:– easy to set up – usable in typical office type room environments without special acoustical treatment– provide a mostly realistic background noise simulation for all typical background noises occurring in telecommunication applications– easy to calibrate – mostly insensitive to the positioning of (test) objects in the simulated sound field– applicable to all typical terminals used in telecommunication– available at reasonable costs ETSI STQ, Taiwan Workshop, February 13th, 2006

5. STF 273 : The Recording Setup • Recordings for the setup in office rooms with • Artificial heads (ITU-T Rec. P.58) or • Binaural probe microphone (MIRE technique) ETSI STQ, Taiwan Workshop, February 13th, 2006

6. t » t » 2m 2m Terminal under test » t Subwoofer Playback from database » t 2m 2m t » Analyser (1/2-channel) STF 273 :The Background Noise Simulation Setup: 4.1 - Technique ETSI STQ, Taiwan Workshop, February 13th, 2006

7. STF 273 : Comparison of Terminal Performance: Simulated vs. Real Background Noise Mobile phone Headset Wideband conference system Desktop HFT Green – original Red - simulated ETSI STQ, Taiwan Workshop, February 13th, 2006

8. STF 273 • Background noise Recording in cars • Instead of an artificial head, two methods for sound pick-up and equalization : • the microphone of the terminal under test, • a pair of cardioid microphones • Background noise Reproduction in cars ETSI STQ, Taiwan Workshop, February 13th, 2006

9. STF 294 Improving the quality of eEurope wideband speech applications by developing a standardised performance testing and evaluation methodology for background noise transmission Objectives : to develop a quality model for wideband speech for new multimedia services (e-health, e-learning, e-bussiness,…) Program in two phases • Phase 1: Network simulation, Subjective tests (in french and csez langages) on realistic scenarios with background noise. Finalized in end 2006; • EG 202 396-2 available • Phase 2 : Development and validation of an objective meyhodology for background noise transmission. Under Progress  August 2006 ETSI STQ, Taiwan Workshop, February 13th, 2006

10. Why Background noise is a problem in mostly all situations and conditions?Both networks and terminals involved • In mobile communication background noise is almost always present; • Problem emphasized due to the acoustical construction of mobile terminals. • Situation even worse for mobile hands-free terminals and mobile car hands-free terminals. • In multimedia conference and classroom systems hands-free type terminal with quite high distance between microphone und speaker. • Computer terminals (e.g. in VoIP scenarios) often used as hands-free terminals • Background noise may be produced by the computer/handheld device itself but may also be picked up from the environment such as an open plan office. • Effects of transmission network impairments in packet based communication networks, Wireless LAN,… ETSI STQ, Taiwan Workshop, February 13th, 2006

11. Scope of the work STF 294 • The aim of the work is to find methods for background noise transmission quality evaluation for real life network scenarios with speech present at the near end and at the far end as well as during periods where no speech is present but only background noise is transmitted. • Steps : • Setup and verification of a transmission network simulation environment using realistic network scenarios for laboratory use; • Setup of a database containing the relevant transmission network models and traffic patterns that would be used for subjective and objective evaluation; • Subjective tests forming the basis for objective evaluation procedures, using the background noise database developed in the currently active ETSI STF 273; • Evaluation and validation of background noise transmission evaluation procedures. ETSI STQ, Taiwan Workshop, February 13th, 2006

12. Phase 1 Tasks – References for the development of the new objective Method • Phase 1 (Leader France Telecom) • Task 1: Transmission network simulation: Telefonica A transmission network simulation environment needs to be setup, to provide comparable network characteristics and traffic patterns as found in conditions of the real transmission networks. • Task 2: Database for transmission networks and conditions evaluation/simulation Telefonica A database containing the typical and realistic transmission network scenarios and traffic patterns needs to be created. There are no risks expected in this task, the procedures are known by the experts in this field. • Task 3: Subjective tests, France Telecom (leader), Mesaqin Subjective test results providing the speech quality parameters in the background noise situation by the individual parameters are to be conducted. This task must be conducted by experts in the field of subjective testing. ETSI STQ, Taiwan Workshop, February 13th, 2006

13. STF 294 ETSI STQ, Taiwan Workshop, February 13th, 2006

14. Task 1 : Transmission network simulation • Evaluation of simulation methodologies and tools • Choice of the best simulation methodologies and tools • Setup of simulation environment in the laboratories of the STF experts • Validation of the simulation environment • The transmission network simulation tool chosen by the experts of STF 294 is an IP network simulator (namely : NIST Net). The network simulation is supposed to provide comparable network characteristics and traffic patterns as found in conditions of the real transmission networks (in terms of packet loss, delay and delay variation distributions). ETSI STQ, Taiwan Workshop, February 13th, 2006

15. Task 2 : Database for transmission networks and conditions evaluation/simulation • Definition of the different network scenarios and conditions to be recorded • Preparation of the simulation recordings • Editing and preparing the data base • Description of the data base (STQ document) ETSI STQ, Taiwan Workshop, February 13th, 2006

16. Step 1 :Clean speech sequences recording Noise-free speech samples (8 samples) Step 2 :Noisy conditions adding Speech samples with background noise conditions (80 samples) Step 3 : Wideband terminal simulation Speech samples with noise and terminal conditions (720 samples) Step 4 : Network simulation Speech sample database (4320 samples) Database of noise type/wideband terminal/network impairment combinations Steps in Recording Process ETSI STQ, Taiwan Workshop, February 13th, 2006

17. Recorded speech samples • For the recordings performed by ETSI STF 294 the following conditions have been used: • 48 kHz (16 bit) sampling rate; • wave format; and • the active speech level has been equalized to -26 dBoV (see ITU-T Recommendation P.56 [24], no filtering applied). • Also for STF 294, the number of samples to have a representative database large enough for subjective and objective evaluation has been decided: • 4 speakers (2 male, 2 female), 8 sentences each; • two languages: Czech and French. • The length of recordings varies between 24 and 73 s. They contain neutral sentences of 2-3s length separated by pauses. Speech activity factor varies between 30 and 60 %. This arrangement allows for easy later editing to any required format. • Simulated terminals : • Mobile Handset in the street • Hands-Free with PC • Hands-Free in car ETSI STQ, Taiwan Workshop, February 13th, 2006

18. Noisy Conditions / Noisy signal Processing • Noisy Conditions • 5 background noise conditions selected: • cafeteria noise; • office room noise; • road noise; • crossroads; and • car noise (car hands‑free at 130 km/h). • Each combination of speech file plus background noise is recorded at two microphone‑loudspeaker positions: • typical handset microphone position (with loudness ratings adjusted to 7 dB); and • hands‑free microphone position (with loudness ratings adjusted to 11 dB). • Noisy signal Processing • Three parameters • With and without VAD (Voice Activity Detection) • 2 different noise reduction filters • 2 different noise reduction levels ETSI STQ, Taiwan Workshop, February 13th, 2006

19. Task 3 : Subjective tests • Preparation of the test material • Conducting experts tests • Evaluation of the experts test results • Preparing test material for naïve listeners tests • Conducting naïve listeners tests (2 languages : French and Czech) • Evaluation of the results • Statistical analysis  the test results ETSI STQ, Taiwan Workshop, February 13th, 2006

20. Listening tests • Method used : according to ITU-T Recommendation P.835 • three questions : • - perception and judgment of noise level reduction, • - perception and judgment of possible distortion on speech signal, • - overall perception of quality. • All these questions must be answered on a discrete 5 level scale. Mesaquin Test Lab France Telecom R&D Test Laboratory ETSI STQ, Taiwan Workshop, February 13th, 2006

21. Exemples of results (1) • MOS distribution and associated 95% CI per noise condition for Czech and French data ETSI STQ, Taiwan Workshop, February 13th, 2006

22. Exemples of results (2) • MOS distribution and associated 95% CI per network condition for Czech and French data (NETWORK I: delay jitter 0ms. packet loss 0%. NETWORK II: delay jitter 10ms. packet loss 1%. NETWORK III: delay jitter 20ms. packet loss 3%) ETSI STQ, Taiwan Workshop, February 13th, 2006

23. Exemples of results (3) • MOS distribution and associated 95% CI per NSA condition ETSI STQ, Taiwan Workshop, February 13th, 2006

24. What do we also find in EG 202 396-2 • Performance parameter descriptions (IP Transmission) • Delay (from codec, packetization, network,…) • Jitter • Packet loss • Parameter interaction and dependences • Wideband speech codecs • Characteristics • Descriptions ETSI STQ, Taiwan Workshop, February 13th, 2006

25. What do we also find in EG 202 396-2 • Wideband speech codecs • Characteristics • Speech coding algorithm • DTX (discontinuous transmission) • Packet Loss Concealment (PLC) • … • Description of some wideband codecs • ITU-T G.722 • AMR WB • … ETSI STQ, Taiwan Workshop, February 13th, 2006

26. Next Phase – Development, Validation and sharing of he new objective Method • Task 4: Objective method An objective method for the quantification of background noise transmission performance needs to be developed. New methods shall be developed and/or existing methods have to be adapted to the problem. • Task 5: Validation of the new method The validation has to be performed independently from the development of the background noise evaluation method based on unknown data acquired in task 3. • Task 6: Final standardization of the new method The final standard(s) are drafted and adapted to the results of the previous tasks. An STQ workshop is planned and realized. • Workshopin Praha, November 13-14, 2006 ETSI STQ, Taiwan Workshop, February 13th, 2006

27. Complementary informations • Visit the STF 273 and STF 294 Web sites • http://portal.etsi.org/STFs/STQ/STF273.asp • http://portal.etsi.org/STFs/STQ/STF294.asp • You will find : • ToRs of the STF • More informations on the work programme • Coordinates of the experts • Database, results,… ETSI STQ, Taiwan Workshop, February 13th, 2006

28. Questions ? ETSI STQ, Taiwan Workshop, February 13th, 2006