1. SEMA SEmantic description of Musical Audio with applications in audio-mining, interactive multimedia, and brain research. Benoit Catteau Prof. LEMAN Marc Faculty of Letters and Philosophy Prof. MARTENS Jean-Pierre Faculty of Applied Sciences

2. SEMA Introduction SEMA The mission My research topics Research resources The auditory model MAMI project Conclusion

3. SEMA Introduction SEMA The mission My research topics Research resources The auditory model MAMI project Conclusion

4. SEMAIntroduction • Music is important in a human’s life • People that have a big music collection need efficient search strategies • Searching for a song now restricted to • Title • Artist • Genre

5. … … … SEMAIntroduction • See music collection as a databaseOwner queries the database • New search algorithms need • Input from the user (textual, auditive imitation) • Automatic annotation of the database Annotations Audio Audio

6. SEMA • Introduction • SEMA • The mission • My research topics • Research resources • The auditory model • MAMI project • Conclusion

7. SEMAThe mission Engineers audio structures Syntactic analysis Statistical models Subjective appreciation Musical experts Syntactic Annotation Semantic Annotation

8. SEMA • Introduction • SEMA • The mission • My research topics • Research resources • The auditory model • MAMI project • Conclusion

9. SEMAMy topics: structures • Extraction of melody lines (bass line, dominant melody, …) and tonality (key) • Reveal the musical structure: chords, harmonic progressions, rythmic patterns, … • Identification of voices (vocal parts, musical instruments)

10. Class I Supervised Learning Algorithm Syntactic Elements Tracks Class … … … SEMAMy topics: semantics • Need for manually annotated databases for training and evaluation of algorithms

11. SEMA Introduction SEMA The mission My research topics Research resources The auditory model MAMI project Conclusion

12. BPF audio Cochlear preprocessing Neural firing patterns SEMAresearch resources: auditory model • Prepocessing the input: give the machine an artificial ear • Time/frequency analysis in peripheral ear

13. SEMAresearch resources: auditory model • Further analysis of neural firing patterns syntactic elements BPF audio Cochlear preprocessing Central auditory processing (???) Ear model Processing module

14. SEMA Introduction SEMA General The mission Research resources The auditory model MAMI project Conclusion

15. SEMAresearch resources: MAMI project • MAMI = “Musical Audio MIning” • Goal: querying the database by humming, whistling, singing, playing the melody (monophonic!) • Method: melody transcription of query, comparing this transcription with MIDI scores of target melodies

16. SEMAresearch resources: MAMI project • Needed central auditory processing pitch pattern BPF audio Cochlear preprocessing Central auditory processing (???) Ear model Processing module

17. SEMAresearch resources: MAMI project • Phase 1 : Time-based algorithm (AMPEX) • developed mainly for speech • limited frequency range (< 400 Hz) • Phase 2 : Frequency-based algorithm (SHS) • analysis of spectral energy distributions • search for harmonics and harmonic ratios • Phase 3 : Combine the two algorithms

18. audio BPF AMPEX Cochlear preprocessing Frequency Splitter Combination pitch Ear model Filtering/Sampling SHS SEMAresearch resources: MAMI project

19. SEMAresearch resources: MAMI project • Results of the pitch detection for singing without lyrics

20. SEMA Introduction SEMA General The mission Research resources The auditory model MAMI project Conclusion

21. SEMAconclusion • We will extend the work of MAMI: • Eliminate the use of MIDI in QBH • Learn how to process polyphonic music • Develop semantic classification algorithms that can handle a wide class of musical pieces • Use these semantic classifications in advanced search methods