1 / 47

Levy Sheet Music Project and Optical Music Recognition introducing Gamut

Levy Sheet Music Project and Optical Music Recognition introducing Gamut. Ichiro Fujinaga McGill University OCVE Workshop (May 2004). Contents. Levy Project Optical Music Recognition Gamera / Gamut Guido / NoteAbility Other Projects. Lester S. Levy Collection. Lester S. Levy Collection.

gezana
Télécharger la présentation

Levy Sheet Music Project and Optical Music Recognition introducing Gamut

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Levy Sheet Music Projectand Optical Music RecognitionintroducingGamut Ichiro Fujinaga McGill University OCVE Workshop (May 2004) OCVE 2004 Fujinaga

  2. Contents • Levy Project • Optical Music Recognition • Gamera / Gamut • Guido / NoteAbility • Other Projects OCVE 2004 Fujinaga

  3. Lester S. Levy Collection OCVE 2004 Fujinaga

  4. Lester S. Levy Collection • North American sheet music (1780–1960) • Digitized 29,000 pieces • including “The Star-Spangle Banner” and “Yankee Doodle” • Database of: • text index records • images of music (8bit gray) • lyrics (first lines of verse and chorus) • colour images of cover sheets (32bit) OCVE 2004 Fujinaga

  5. Digital Workflow Management • Reduce the manual intervention for large-scale digitization projects • Creation of data repository (text, image, sound) • Optical Music Recognition (OMR) • Gamera / Gamut • XML-based metadata • composer, lyricist, arranger, performer, artist, engraver, lithographer, dedicatee, and publisher • cross-references for various forms of names, pseudonyms • authoritative versions of names and subject terms • Music and lyric search engines • Music analysis toolkit OCVE 2004 Fujinaga

  6. Optical Music Recognition (OMR) • Open-source adaptive OMR system in development since 1984 • Staff recognition and removal • Run-length coding • Projections • Lyric removal • Stems and notehead removal • Music symbol classifier (trainable) • Score reconstruction OCVE 2004 Fujinaga

  7. OMR: Classifier • Connected-component analysis • Feature extraction, e.g.: • Width, height, aspect ratio • Number of holes • Central moments • k-nearest neighbor classifier • Genetic algorithm OCVE 2004 Fujinaga

  8. Overall Architecture for OMR Image File Staff removal Segmentation Recognition K-NN Classifier Output Symbol Name Optimization Genetic Algorithm K-nn Classifier Knowledge Base Feature Vectors Best Weight Vector Off-line OCVE 2004 Fujinaga

  9. The problem • Suitable OCR for lyrics not found • Commercial OCR systems are often inadequate for non-standard documents • The market for specialized recognition of historical documents is very small • Researchers performing document recognition often “re-invent” the basic image processing wheel OCVE 2004 Fujinaga

  10. The solution • Provide easy to use tools to allow domain experts (people with specialized knowledge of a collection) to create custom recognition applications • Generalize OMR for structured documents OCVE 2004 Fujinaga

  11. Created Gamera • Framework for creation of structured document recognition system • Designed for domain experts • Image processing tools (filters, binarizations, etc.) • Document segmentation and analysis • Symbol segmentation and classification • Feature extraction and selection • Classifier selection and combiners • Syntactical and semantic analysis Generalized Algorithms and Methods for Enhancement and Restoration of Archives OCVE 2004 Fujinaga

  12. Features of Gamera • Portability (Unix, Windows, Mac) • Extensibility (Python and C++ plugins) • Easy-to-use (experts and programmers) • Open source • Graphic User Interface • Interactive / Batchable (scripts) OCVE 2004 Fujinaga

  13. Architecture of Gamera Scripting Environment (Python) Graphic User Interface (wxWindows) Plugins (Python) Plugins (C++) GAMERA Core (C++) OCVE 2004 Fujinaga

  14. Example of C++ Plugin // Number of pixels in matrix Class area(PluginFunction): self_type = ImageType([ALL]) return_type = FloatVector(“area”, 1) #include “gamera.hpp” using namespace Gamera; template <class T> feature_t area(T &m) { return feature_t(m.nrows() * m.ncols()); } OCVE 2004 Fujinaga

  15. Example of Python function // This filters a list of CC objects from gamera.core import * def filter_wide(ccs, max_width): tmp = [] for x in ccs: if x.ncols() > max_width: x.fill_matrix(0) else: tmp.append(x) return tmp OCVE 2004 Fujinaga

  16. Gamera: Interface(screenshot in Linux) OCVE 2004 Fujinaga

  17. Gamera: Interface(screenshot in Linux) OCVE 2004 Fujinaga

  18. Histogram(screenshot in Linux) OCVE 2004 Fujinaga

  19. Thresholding(screenshot in Linux) OCVE 2004 Fujinaga

  20. Thresholding(screenshot in Linux) OCVE 2004 Fujinaga

  21. OMR reborn as Gamut • Built within the Gamera framwork • Designed by domain experts • Lyric separtation and recognition • Staffline removal routine • Stems and notehead removal • Music symbol segmentation and classification (trainable) • Score reconstruction • Guido (NoteAbility) • Coming soon… MusicXML (Finale, Sibelius) Gamera-basedAdaptiveMusicUnderstandingTools OCVE 2004 Fujinaga

  22. Staff removal: Lute tablature OCVE 2004 Fujinaga

  23. OCVE 2004 Fujinaga

  24. Classifier: Lute(screenshot in Linux) OCVE 2004 Fujinaga

  25. Staff removal: Neums OCVE 2004 Fujinaga

  26. Classifier: Neums(screenshot in Linux) OCVE 2004 Fujinaga

  27. Greek example OCVE 2004 Fujinaga

  28. “A formal language for score-level representation” Plain text: readable, platform independent Extensible and flexible Adequate representation NoteServer: Web/Windows GUIDO/XML NoteAbility (K. Hamel) GUIDO Music Notation FormatH. Hoos, K. Renz, J. Kilian OCVE 2004 Fujinaga

  29. GUIDO: An example { [ \beamsOff | \clef<"treble"> \key<"D"> f#*1/8. g*1/16 | a*1/4. d2*1/8 d*1/4. c#*1/8 | e1*1/2 _*1/4 f#*1/8. g*1/16 | c#2*1/4. b1*1/8 a*1/4. g*1/8 | | e#*1/2 f#*1/4 f#*1/8. g*1/16 | a*1/4. d2*1/8 d*1/4. c#*1/8 | e1*1/2 _*1/4 f#*1/8 g | c#2*1/4. b1*1/8 a*1/4. c#*1/8 ], . . . OCVE 2004 Fujinaga

  30. NoteAbility Demo OCVE 2004 Fujinaga

  31. Summary of Gamera • Gamera allows rapid development of domain-specific document recognition applications • Domain experts can customize and control all aspects of the recognition process • Includes an easy-to-use interactive environment for experimentation • Beta version available on sourceforge.net for Linux, OS X, and Windows OCVE 2004 Fujinaga

  32. Recent Developments • Sheet Music Consortium • OAI (Open Archive Initiative) • Library of Congress • Chopin Early Editions • MODS (Metadata Object Description Schema) • METS (Metadata Encoding and Transmission Standard) • University of Maine OCVE 2004 Fujinaga

  33. Sheet Music Consortium OCVE 2004 Fujinaga

  34. Sheet Music Consortium OCVE 2004 Fujinaga

  35. Library of Congress OCVE 2004 Fujinaga

  36. Chopin Early Editions OCVE 2004 Fujinaga

  37. Chopin Early Editions OCVE 2004 Fujinaga

  38. Maine Music Box OCVE 2004 Fujinaga

  39. TowardsDistributed Digital Music Archives and Libraries(DDMAL) • Open Standards • Open Source • Open File Formats (non-binary) • XML (MODS, METS) • Web Services • UDDI (Universal Description, Discovery, and Integration) • SOAP (Simple Object Access Protocol ) • WSDL (Web Services Description Language) OCVE 2004 Fujinaga

  40. Conclusions • Levy Project • Optical Music Recognition • Gamera /Gamut • Guido / NoteAbility • Other Projects • Distributed Digital Music Archives and Libraries (DDMAL) OCVE 2004 Fujinaga

  41. Acknowledgments • Michael Droettboom (Gamera / Gamut) • Karl McMillan (Gamera) • Robert Ferguson (OS X port) • Keith Hamel (NoteAbility) • JHU Digital Knowledge Center • National Science Foundation • Québec Government • Levy Family OCVE 2004 Fujinaga

  42. OCVE 2004 Fujinaga

  43. Projections X-projections Y-projections back OCVE 2004 Fujinaga

  44. Chopin Early Editions OCVE 2004 Fujinaga

  45. Chopin Early Editions OCVE 2004 Fujinaga

  46. Chopin Early Editions OCVE 2004 Fujinaga

  47. Chopin Early Editions OCVE 2004 Fujinaga

More Related