Developing a Virtual Piano Playing Environment

1. Developing a VirtualPiano Playing Environment By combining distributed functionality among independent Agents A.Broersen

2. Overview • Introduction to the project • Backgrounds of the concepts • Design of a multi-agent platform • Playing with the system • Conclusions and remarks • Work in progress towards the aid for a real piano teacher

3. Introduction: Parlevink (http://parlevink.cs.utwente.nl) A group on Language, Knowledge and Interaction doing research on: • Human-Computer Interaction • Dialogue systems • Autonomous agents • Virtual worlds Overview Introduction Design Playing Conclusions Work in progress Demonstration

4. Introduction: Field Exploring possibilities usingVR and Human-Computer Interactionin distributed learning environments. Case: Piano Playing • interactions using a real keyboard, VR, dialogues and dataglove • playing with more people at the same time • creating musical tools and a teacher in a dynamic learning environment Overview Introduction Design Playing Conclusions Work in progress Demonstration

5. Introduction: ADRI (Artificial Didactic Recital Instructor) GoalBuilding a framework for a piano teacher in a virtual piano playing environment With • a real keyboard connected to • a piano in a 3D-world with • tools for music- and piano-education • in a distributed architecture Overview Introduction Design Playing Conclusions Work in progress Demonstration

6. Introduction: Requirements ADRI should contain: • different independent processes • server functionality • Java3D • Midi for sound • using Windows and Unix • general knowledge/lesson structure Overview Introduction Design Playing Conclusions Work in progress Demonstration

7. Design: Agents For flexibility different ‘Agents’ are used What are Agents in ADRI? • Different functions different skilled agents • When can you call the processes ‘Agents’? • When are agents ‘Intelligent’? Overview Introduction Design Playing Conclusions Work in progress Demonstration

8. Design: Multi-Agent Platform Connecting the agents with each other in a platform: • Simple communication (messages) between agents • On each machine one agent to control the ADRI-agents • One agent is manually controlled to handle the available agents and their different connections Overview Introduction Design Playing Conclusions Work in progress Demonstration

9. Design: Communication Exchanging XML-messages: • Communication messages using a simple protocol • Midi messages as quickly as possible • ADRI messages exchange of knowledge (graphs)using multicast functionality ‘Is someone listening?’ Overview Introduction Design Playing Conclusions Work in progress Demonstration

10. Playing: The Agents • Midiator playing and capturing Midi sounds • Visualizer user dialogue in a 3D World with a visualization of a piano and notes • Supervisor (remotely) control the system • Communicator handle agent functionalityon one Java Virtual Machine Overview Introduction Design Playing Conclusions Work in progress Demonstration

11. Playing: More Agents • Expert provide songs and chord recognition • Analyser analyses the notes in a song • Observer tracks the users actions and progress • Instructor gives assignments to a learner Overview Introduction Design Playing Conclusions Work in progress Demonstration

12. Playing: User Interface In the Supervisor: • Dynamic visualization and control of the complete agent network in a graph • An agent dialogue frame with the specific options in the Agent • Sending messages to agents Different combinations of agents give different environments Overview Introduction Design Playing Conclusions Work in progress Demonstration

13. Playing: The Visualizer

14. Conclusions: Agents In the resulting prototype: • the agents are kept ‘simple’ • the graphical Supervisor provides a good control and overview of the system • the message-parsing is fast enough for real-time exchange of Midi • there is no good structure (yet) for knowledge and lesson structure Overview Introduction Design Playing Conclusions Work in progress Demonstration

15. Conclusions: Communication Using no standard package: • The communication layer is still under development according to different needs in the application • Quick exchange of sound and graphical events using Midi • Midi capture is a problemusing only Java Overview Introduction Design Playing Conclusions Work in progress Demonstration

16. Work in progress: Building... • Creating more demo Agents: • Navigator • Translator • Corrector • Using advise and remarks of human experts and users • Developing plans for lessons:Using graphs for state-transition diagrams Overview Introduction Design Playing Conclusions Work in progress Demonstration

17. Work in progress: Where? For (more) information or questions: • Parlevink, Dept. of Computer Science • University of Twente, the Netherlands • http://parlevink.cs.utwente.nl • broersen@cs.utwente.nl Overview Introduction Design Playing Conclusions Work in progress Demonstration

18. Demonstration: Playing

19. Demonstration: Dataglove

20. Playing: The Supervisor

21. Demonstration: Screen-shots Host address Remote Host Route to Supervisor Other Agent ADRI Agent

22. Demonstration: Screen-shots Terminator Multicast Member Multicast Group Communication Group

23. Playing: The Midiator