Research themes Coordinated Multi Modal Generation for Virtual Humans

1. The Virtual Conductor Research themes Coordinated Multi Modal Generation for Virtual Humans Coordination of multi modal generation with perception input. Real-time coordinated interaction. Dynamic animation, synchronized with extrapolations of a stream of incoming events/observations. Conversational Agents: Theoretical Work The Virtual Conductor will serve as a context for theoretical investigation of dialog as a continuous perception/production feedback loop in which both interaction parties are at all times 'sending' as well as 'receiving'. Conducting Applications The Virtual Conductor (VC) may be used for several applications. The most relevant at this point are the use of the VC for rehearsals and for tutoring of conducting students. Music Perception Technology The VC presents a novel context to work on applications of music analysis technology. Activities Analyzing Conducting by Humans To implement the right behavior in a VC we need to know how different intentions are expressed by human conductors, and when. Music Perception Technology To conduct effectively, the VC needs to analyze the music in real time using algorithms implemented and/or adapted from literature. Multi Modal Generation Animation technology, adaptive (re)planning, synchronization, etc… Evaluation with Human Musicians Conduct tests with human musicians to evaluate the interaction models and the perception technology. • Current State • Animated Virtual Conductor • Conducts a group of human musicians • Using 1-, 2-, 3- and 4-beat patterns. • Conducts volume and tempo (changes). • Tempo interaction: prepares, and leads the musicians through, tempo changes or corrections. • Can rehearse problematic passages with the ensemble • The conductor has been evaluated several times with groups of human musicians. The musicians could follow the tempo and dynamic changes of the conductor reasonably well. The conductor could interact successfully with the musicians, correcting their tempo if they played too fast or to slow. Human Media Interaction P. Bos A. Nijholt D. Reidsma Zs. Ruttkay d.reidsma@ewi.utwente.nl • Interacting with a Virtual Conductor, Bos, P., Reidsma, D., Ruttkay, Z.M., and Nijholt, A. (2006) in: Proc. of 5th International Conference on Entertainment Computing, Cambridge, UK. pp 25-30, LNCS 4161 Springer Verlag, DOI 1007/11872320_3 • http://hmi.ewi.utwente.nl/showcases/Human Music Interaction/