Download
1 / 29
290 likes | 419 Vues
This research focuses on efficient audio rendering for complex scenes with moving sources, aiming to maintain high audio quality without audible impairments. Utilizing a recursive clustering approach, the method enhances scalable perceptual premixing while taking into account perceptual and cross-modal information. Conducted user studies validate the performance improvements in audio quality, enabling the treatment of up to 3000 sound sources. The study provides insights into the effectiveness of various clustering techniques and the influence of cross-modal effects on auditory perception.
E N D
Progressive Perceptual Audio Rendering of Complex Scenes Thomas Moeck - Nicolas Bonneel - Nicolas Tsingos - George Drettakis - Isabelle Viaud-Delmon - David Alloza 1 4 1,2 1 3 1 1- REVES/INRIA Sophia-Antipolis 2- Computer Graphics Group, University of Erlangen-Nuremberg 3- CNRS-UPMC UMR 7593 4- EdenGames
Objectives • Efficient audio rendering of very complex scenes with moving sources • Without audible impairment of the quality • Verify results by user tests
Previous Work • Rendering complex auditory scenes • Clustering [Tsingos et al. 2004]: replace many sources with a representative • Still can only treat ~200 sound sources (cost of clustering itself) • Scalable audio processing • Importance-guided processing of few frequency/time bins [Fouad et al. 1997, Wand & Straßer 2004, Gallo et al. 2005, Tsingos 2005]. • Audio processing (e.g., HRTF, spatialization) is expensive • Crossmodal effects • Neuroscience Literature: “Ventriloquism affects 3D audio perception” • Ventriloquism spatial window can vary from a few up to 15 degree • Few papers on ecological experiments
Methodology • Recursive approach to clustering • Reduce cost of clustering • Scalable perceptual premixing • Faster premixing without audible loss of quality • Taking perceptual and cross-modal information into account • Improve audio clustering algorithm • User experiments to detect improvement possibilities • Improving quality with results of tests • Validation of resulting algorithms
Overview of the algorithms • Masking of inaudible sources (with energy) • Clustering of remaining sources • Progressive premixing within each cluster • Spatial audio processing (HRTF) recursive
Our Work • Optimized recursive approach of clustering • Clustering performance evaluation • Improved scalable perceptual premixing • Quality evaluation study • Study of cross-modal effects by user experiments • Using results of cross-modal studies to develop audio-visual clustering algorithm
Optimized Recursive Clustering • Recursive splitting of clusters • Fixed-budget approach • Using a fixed number of clusters • Variable-budget approach • Splitting clusters until break condition is reached • Break condition: Average angle error • Optimal number of clusters • Variant used by EdenGames • 8 cluster budget • Local clustering when necessary
Clustering Performance Evaluation • Performance of recursive algorithms are clearly better
Improved progressive scalable perceptual premixing (1) • After clustering: Premixing in each cluster • Why? Effects can be done afterwards - less cost because viewer signals • Only premixing necessary data • Assigning frequency bins to sound sources (iterative importance sampling) by using pinnacle value
Improved progressive scalable perceptual premixing (2) premixing clustering
Improved progressive scalable perceptual premixing (3) • Iterative importance sampling • Calculation of importance value from energy, loudness or audio saliency map • Assignment of frequency proportional to importance • until pinnacle value is reached • Reassignment of remaining frequencies to sounds relative to importance values
Quality Evaluation Study (1) • MUSHRA (“Multiple Stimuli with Hidden Reference and Anchors”) test of perceptual premixing • 7 subjects, aged from 23 – 40 • Ambient, music and speech • Various budgets (2% – 25 %) • With and without pinnacle value • Using loudness or saliency as importance value
Quality Evaluation Study (2) • Results: • Approach is capable of generating high quality using 25% of the original data • Acceptable results with 10% (2% in case of speech) • Significant Effects: • Budget • Importance value • Pinnacle value
Study of Cross-Modal Influences – Questions • Do we need more or fewer clusters in the viewing frustum? • We move spatial position of sound sources to representative in cluster • How tolerant are we to this error ? • Do visuals influence the perceived quality?
Study of Cross-Modal Influences – Results • Statistical analysis of the results shows: • We need more clusters in the viewing frustum • No significant difference of visuals/no-visuals but possible cross-modal effect
Modifying the algorithm • Introducing weighting term in clustering: Increasing number of clusters in the viewing frustum
Conclusions • Up to nearly 3000 sound sources possible in good quality • Main limitation are graphics (!) • Better quality because more clusters in viewing frustum • Future work • experiment with auditory saliency measurements • handle procedurally synthesized sounds?
