Jürgen Scheible , Media Lab, University of Art and Design Helsinki, Finland

Wireless Cities 2006 Conference5th - 8th June, Oulu Finland Multimodal Interfaces for Mobile Multimedia Jürgen Scheible, Media Lab, University of Art and Design Helsinki, Finland © 2006, Jürgen Scheible

Outline • Artistic motivation & Background • Related work & Research challenges • MobiLenin – Combining A Multi-Track Music Video,Personal Mobile Phones and A Public Display into Multi-User Interactive Entertainment • Experimental evaluation • Research findings • Conclusions & Future work © 2006, Jürgen Scheible

Artistic Motivation & Background • Music artist (Lenin’s Godson) • Pop/rock • Compose, sing, rap, play guitar, produce, live performances • Media artist • Vj-ing • Interactive music video • Computer games • Mobile art • Diplom-Engineer (telecommunications) with8 years of work experience in Nokia • Since 2003 Doctoral Student at University of Art and Design Helsinki / Media Lab © 2006, Jürgen Scheible

Artistic Motivation & Background • My approach: draw ideas from my artistic approaches and turn them into technical solutions / research projects Proof of concept by rapid prototyping Empirical evaluation with genuine end users in the true environment of use • Feedback for the design of new appealing and original art pieces and for advancing the research field © 2006, Jürgen Scheible

Artistic Motivation for MobiLenin • Create interactive system for virtual interaction with the artist • To enhance people's concert experience by exceeding the limitations of the physical domain • E.g. turn the artist into a skeleton • “Holistic” contribution into an interactive art piece • Own music and producing the music video • Being myself the actor / performer in the video • System and interaction design • Implementation • Earlier installations: Interaction with a music video by • Measuring the volume of shouting and clapping of an audience • Camera-based tracking of a large green coloured ball filled with air thrown into the audience © 2006, Jürgen Scheible

Related work Audience participation systems where people are able to control onscreen activity by: • Leaning left and right in their seats to steer a race car or move a paddle in the video game Pong (Video Image Analysis) • Batting a beach ball while its shadow is used as a pointing device • Pointing laser pointers at the screen • Cinematrix Interactive Entertainment System: audience members can participate interactively in activities such as maze navigation and opinion polling by displaying the red or the green side of a paddle (See paper for references) © 2006, Jürgen Scheible

Research challenges • The greatest challenge does not lie in developing technology for audience interaction, but in designing engaging activities(Maynes-Aminzade et al. 2002) • Half the battle is designing how the display will invite interaction (Agamanolis 2003) • Users need constant encouragement and demonstration to interact with the interactive public display (Churchill et al. 2004) • There is little research on how situating displays in different places invites certain kinds of social interactions (Rogers & Lindley 2004) • Most research on interaction techniques focuses on single users or on small groups (Maynes-Aminzade et al. 2002) © 2006, Jürgen Scheible

 Show video The MobiLenin system • Change performance style of artist (6 styles) by voting • Group of people interact with music video on large public display using their personal mobile phones for enriched entertaining and social experience • Hybrid interface combining complementary strengths of a public display and personal mobile phones to overcome their respective weaknesses • Public display  Great conceptual power and large presentation space  Typically limit interaction to one user at time • Personal mobile phones  Limited conceptual power and screen size  Disperse control and access • Lottery mechamism as an incentive for interaction © 2006, Jürgen Scheible

System architecture Simple client-server architecture • Mobile clientprogrammed in Python for Series 60 • Each client is connected to the server via HTTP over GPRS 2. Serverprogrammed with Macromedia Director • Controls the QuickTime player with multi-track video • Controls client UI • Counts votes / analysis • Conducts lottery mechanism • Initiates the delivery of winning notifications External server component consisting of simple PhP scripts • Mediator between public mobile data network and PC 3. Large public display • Main user interface • Shows the interactive video & result bar & winning notification  Server-driven real-time control of clients withprivate return channel © 2006, Jürgen Scheible

Multi-Track Music Video Production of the multi-track music video: • Foreground video for the six tracks was shot in a studio against a blue screen • Each performing styles filmed separately with full length of the song • The skeleton - was produced via a stop motion animation using a plastic toy skeleton • Background (busy street) video added via bluescreening • 6 parallel video tacks (inserted in 1 quicktime movie) • 3 parallel music tracks (inserted in same quicktime movie) © 2006, Jürgen Scheible

Experimental user evaluation With genuine users in true environment of use • 14 test users • 8 males, 6 females, age 18-49 years • Qualitative data reflecting the user experience • Questionnaire (21 statements + 14 open-ended questions) • Observation during the experiment (3 video cameras) • Video interviews after the experiment • Quantitative data via logging in the server © 2006, Jürgen Scheible

Research findings A very strong general observation: People enjoyed using the MobiLenin system • This was expressed by laughing, happy faces, good mood, and rowdy celebrations upon winning in the lottery Let’s look in detail at: • Interaction / system design • Social setting • Role of personal mobile phone in interaction • Interactive content © 2006, Jürgen Scheible

Research findings:Interaction / system design • Ease of use “The system was easy to use” (10/2) • Clearness of what the system has to offer “I felt it was clear from beginning what the system has to offer and how to interact ”(9/4) • Constant but light feedback “I found the popup notes useful on the phone”(12/0) • Multiple choices in interacting “Better only up down or stay?“ (0/11) • Lottery mechanism “Idea of having a lottery to win a pizza or a beer in such a system is good”(13/0) • The system enticed interaction with the display ”I enjoyed the interaction”(11/2) • The system stimulated inter-personal social interaction “Better if I would have been alone to interact with the screen”(0/12) Notation: “Statement” (agree / disagree) © 2006, Jürgen Scheible

Research findings: Social setting • Co-location • Face-to-face contact • Awareness of what others in the co-located group do • Private and public GUIs to foster social dynamics • Spectator view • Drop in drop out © 2006, Jürgen Scheible

Research findings:Role of personal mobile phone in interaction • Personal trusted device – feel uninhibited to use • Ubiquitousness - remove need for any additional, specific user devices • Disperses access and control, facilitating multi-user interaction with a single public display • Allows anonymous participation in group activity (feel invited & uninhibited) • Guarantees each user equal power in the interaction • Provides reliable return channel for delivering confidential user specific information back to the user © 2006, Jürgen Scheible

Research findings: Interactive content Key lies in the production of content, so that the user is provided with sufficiently rich set of choices to choose from at different times, so that (s)he can anticipate “something new” at the point of voting and when the vote is revealed, to experience “something new”. • "Connected with interests of people e.g. different music styles, pop, rock etc. so people could find and see what bunch of people you have there, who likes what.“ • "Avatar, lot of different properties to construct the avatar, would be interesting to see if a group could construct it, so to change features to other parts of the avatar or environment. • Properties of funny things coming into the film."; "Found it very funny to have the skeleton, it gave nice picture of innovativeness and humor.“ • "Building up something gradually would be interesting."; "Imagine night club, vote for next song and video and interact with that.”; “Trivia game to vote answer with group.“ • "Music videos are nice, that inspires me." © 2006, Jürgen Scheible

Conclusions MobiLenin successfully addresses many of the research challenges presented in the literature • Allows multi-user interaction with a single public display • Entices users to interact • Provides an engaging activity • Provides enriched entertaining and social experience © 2006, Jürgen Scheible

Conclusions • MobiLenin architecture provides a framework for realizing similar installations with different types of multimedia content • MobiLenin provides a new form of interactive entertainment for pubs and other public places • MobiLenin is • Scalable (even for large groups, indoors, outdoors, short range, long range) • Flexible (content can be dynamic, since client server driven) • Easy to deploy © 2006, Jürgen Scheible

Future work • Scalability of MobiLenin allows dynamic setting of a live concert • Simple majority votes for the next song to be performed • Complex issues like allowing the audience to interact with the parameters of individual instruments played by the musicians on stage • Using different content and modifying the script of the interaction • Interactive storytelling, trivia, multi-player gaming etc. • Feedback channel provided by the private GUI of the mobile phone application allows addressing individual users or subgroups E.g. system could ask only one fourth of the audience to • Wave their hands in the air • Hug their neighbors • Shake their hands • Shout as loud as possible for the purpose of triggering events full of surprises, something new that the participants are expecting from the interaction © 2006, Jürgen Scheible

Future work • We believe that new and exciting forms of interactive art can emerge by further developing hybrid virtual interfaces such as the MobiLenin system! • Many fun game type of ideas can easily be generated and implemented! • As an artist, engineer and researcher, I want to further contribute to the research community and to inspire the artistic community – by iterating MobiLenin © 2006, Jürgen Scheible

Jürgen Scheible , Media Lab, University of Art and Design Helsinki, Finland

Jürgen Scheible , Media Lab, University of Art and Design Helsinki, Finland

Presentation Transcript