ACE: A Framework for optimizing music classification

1. ACE: A Framework for optimizing music classification Cory McKay Rebecca Fiebrink Daniel McEnnis Beinan Li Ichiro Fujinaga Music Technology Area Faculty of Music McGill University

2. Goals • Highlight limitations of existing pattern recognition software when applied to MIR • Present solutions to these limitations • Stress importance of standardized classification and feature extraction software • Ease of use, portability and extensibility • Present the ACE software framework • Uses meta-learning • Uses classification ensembles 2/25

3. Existing music classification systems • Systems often implemented with specific tasks in mind • Not extensible to general tasks • Often difficult to use for those not involved in project • Need standardized systems for a variety of MIR problems • No need to reimplement existing algorithms • More reliable code • More usable software • Facilitates comparison of methodologies • Important foundations • Marsyas (Tzanetakis & Cook 1999) • M2K (Downie 2004) 3/25

4. Existing general classification systems • Available general-purpose systems: • PRTools (van der Heijden et al. 2004 ) • Weka (Witten & Frank 2005) • Other meta-learning systems: • AST (Lindner and Studer 1999) • Metal (www.metal-kdd.org) 4/25

5. Problems with existing systems • Distribution problems • Proprietary software • Not open source • Limited licence • Music-specific systems are often limited • None use meta-learning • Classifier ensembles rarely used • Interfaces not oriented towards end users • General-purpose systems not designed to meet the particular needs of music 5/25

6. Special needs of music classification (1) • Assign multiple classes to individual recordings • A recording may belong to multiple genres, for example • Allow classification of sub-sections and of overall recordings • Audio features often windowed • Useful for segmentation problems • Maintain logical grouping of multi-dimensional features • Musical features often consist of vectors (e.g. MFCC’s) • This relatedness can provide classification opportunities 6/25

7. Special needs of music classification (2) • Maintain identifying meta-data about instances • Title, performer, composer, date, etc. • Take advantage of hierarchically structured taxonomies • Humans often organize music hierarchically • Can provide classification opportunities • Interface for any user 7/25

8. Standardized file formats • Existing formats such as Weka’s ARFF format cannot represent needed information • Important to enable classification systems to communicate with arbitrary feature extractors • Four XML file formats that meet the above needs are described in proceedings 8/25

9. The ACE framework • ACE (Autonomous Classification Engine) is a classification framework that can be applied to arbitrary types of music classification • Meets all requirements presented above • Java implementation makes ACE portable and easy to install 9/25

10. ACE and meta-learning • Many classification methodologies available • Each have different strengths and weaknesses • Uses meta-learning to experiment with a variety of approaches • Finds approaches well suited to each problem • Makes powerful pattern recognition tools available to non-experts • Useful for benchmarking new classifiers and features 10/25

11. Feature Extraction System Model Classifications Extracted Features Feature Settings Taxonomy Music Recordings Trained Classifiers Statistical Comparison of Classification Methodologies Experiment Coordinator Classification Methodology 1 Classification Methodology n ACE Dimensionality Reduction Dimensionality Reduction … Classifier Evaluator 11/25

12. Algorithms used by ACE • Uses Weka class libraries • Makes it easy to add or develop new algorithms • Candidate classifiers • Induction trees, naive Bayes, k-nearest neighbour, neural networks, support vector machines • Classifier parameters are also varied automatically • Dimensionality reduction • Feature selection using genetic algorithms, principal component analysis, exhaustive searches • Classifier ensembles • Bagging, boosting 12/25

13. Classifier ensembles • Multiple classifiers operating together to arrive at final classifications • e.g. AdaBoost (Freund and Shapire 1996) • Success rates in many MIR areas are behaving asymptotically (Aucouturier and Pachet 2004) • Classifier ensembles could provide some improvement 13/25

14. Musical evaluation experiments • Achieved a 95.6% success with a five-class beatbox recognition experiment (Sinyor et al. 2005) • Repeated Tindale’s percussion recognition experiment (2004) • ACE achieved 96.3% success, as compared to Tindale’s best rate of 94.9% • A reduction in error rate of 27.5% 14/25

15. General evaluation experiments • Applied ACE to six commonly used UCI datasets • Compared results to recently published algorithm (Kotsiantis and Pintelas 2004) 15/25

16. Results of UCI experiments (1) 16/25

17. Results of UCI experiments (2) • ACE performed very well • Statistical uncertainty makes it difficult to say that ACE’s results are inherently superior • ACE can perform at least as well as a state of the art algorithm with no tweaking • ACE achieved these results using only one minute per learning scheme for training and testing 17/25

18. Results of UCI experiments (3) • Different classifiers performed better on different datasets • Supports ACE’s experimental meta-learning approach • Effectiveness of AdaBoost (chosen 2 times out of 6) demonstrates strength of classifier ensembles 18/25

19. Feature extraction • ACE not tied to any particular feature extraction system • Reads Weka ARFF as well as ACE XML files • Does include two powerful and extensible feature extractors are bundled with ACE • Write Weka ARFF as well as ACE XML 19/25

20. jAudio • Reads: • .mp3 • .wav • .aiff • .au • .snd 20/25

21. jSymbolic • Reads MIDI • Uses 111 Bodhidharma features 21/25

22. Graphical interface Includes an on-line manual Command-line interface Batch processing External calls Java API Open source Well documented Easy to extend ACE’s interface 22/25

23. Current status of ACE • In alpha release • Full release scheduled for January 2006 • Finalization of GUI • User constraints on training, classification and meta-learning times • Feature weighting • Expansion of candidate algorithms • Long-term • Distributed processing, unsupervised learning, blackboard systems, automatic cross-project optimization 23/25

24. Conclusions • Need standardized classification software able to deal with the special needs of music • Techniques such as meta-learning and classifier ensembles can lead to improved performance • ACE designed to address these issues 24/25

25. Web site: • coltrane.music.mcgill.ca/ACE • E-mail: • cory.mckay@mail.mcgill.ca