Enhancing Information Awareness Through Speech Induced Anthropomorphism

1. Enhancing Information Awareness Through Speech Induced Anthropomorphism PhD Examination Nuno Jorge Gonçalves de Magalhães Ribeiro External examiner: Professor Philip Barker Internal examiner: Dr. Alistair Edwards Department of Computer Science University of York 06 February 2002

2. Structure of the Examination(follow presentation, 4 blocks, discussion at the end of each block) PhD Examination

3. Motivation Problems with current interfaces for information awareness (lack of humanized interaction) • Finding visually hidden information is distracting (e.g. information hidden behind opened windows, dynamic information) • Lack of semantics in alerting cues (e.g. blinking icons, sounds), or visually intrusive displays (e.g. windows that obscure info. being accessed) • Lack of awareness and co-ordination between high-priority and unimportant alerts to be reported (e.g. junk e-mail, status info.) • Lack of a unified interruption mechanism for notifications (imminent and just occurred events) and reminders (past, but relevant, events) PhD Examination

4. How to Humanize Information Awareness? • One way is to exploit speech to deliver alerting messages (notifications and reminders) BUT (because listening to speech cues anthropomorphic feelings) • The reception of speech demands that the interface reflects human social norms • at the delivery level – design should humanize the delivery of speech-based messages • at the content level – design should humanize the contents of speech-based messages • 3 questions: Why speech? How to humanize the delivery? How to humanize the content? PhD Examination

5. Why Speech? There is an urgent need to find out the situations in which speech enhances the user interface and the situations in which it is detrimental to the user interface This thesis suggests that speech can be used with advantages to: • enable less disruptive interruptions • convey more meaningful notifications • design of an integrated notification mechanism PhD Examination

6. How to Humanize Message Delivery?(in order to reinforce social acceptability of the speech interface) This thesis suggests that there are 3 fundamental design aspects: • Attentive interruptions (minimise user disruption) • When to interrupt: precedence <= prioritiy and urgency, • How to interrupt: get attention, modulate the audio channel when it is already being used • Linguistic variation (avoid irritating repetitive utterances while maintaining the semantics) • Sources: lexical, syntactical, phonetic • Politeness (attenuate irritating effect of interruptions) • Simple courtesy (e.g. Excuse me, I’m sorry) • Design based on politeness strategies (e.g. direct, positive, negative) PhD Examination

7. How to Humanize Message Contents?(in order to enhance the awareness of information that is hidden from view) This thesis suggests that there are 2 fundamental design aspects: • Short alerting messages: just relevant content (because speech is a serial channel) • Context-based cues: extracted from items (because context plays a major role in recalling information) • Forward Cues: Kind, Time, Quantity, People, Nature, Description, Place PhD Examination

8. End of block 1 Questions Discussion

9. Humanized Experimental EnvironmentIn presenting the design, the thesis illustrates: • How to splita complex environment into a number of separate software agents that perform specific monitoring tasks. • How to develop a software architecture that is based on a collection of specialised software agents. • How to design the internal architecture of each software agent, in such a way that it collects and conveys context information. • How to communicate collected information between the software agents comprising the environment. • How to transform collected information into appropriate spoken messages to be delivered to the user. • How to deliver the spoken information following typical social norms. PhD Examination

10. Experimental Environment Architecture PhD Examination

11. Steps in the Humanization Process • Deciding when to interrupt (hum. delivery)(specialised agents: I.L. <= urgency, priority, user presence) • Extracting context cues (hum. content)(specialised agents: forward cues) • Including linguistic variation (hum. content and delivery)(speaking agent: templates for lexical and syntactical variations; prosody markers for phonetic variation) • Including politeness (hum. delivery)(speaking agent: attention-grabbers, templates for linguistic style) • Deciding how to interrupt (hum. delivery)(speaking agent: attention-grabbing, audio channel modulation) PhD Examination

12. Experimental Environment Prototype • Windows environment (familiar, limitations) • Implements specialised agents: • Environmental awareness, context extraction, message composition • E-mail (MAPI), Diary (TXT), Printer (Default printer queue) • Implements communication protocol: • Comprises purposefully designed message types and contents • Relies on transport mechanism based on Dynamic Data Exchange (DDE) • Implements speaking agent: • MS Agent client • Creates spoken reminders and notifications including linguistic variation and politeness • Interrupts by grabbing the userattention, fading audio channel • Uses TTS synthesiser and Speech Recognition engine PhD Examination

13. End of block 2 Questions Discussion

14. Empirical Study: Objectives (1/3) Assess user´s perceptions of the adequacy of anthropomorphic behaviours PhD Examination

15. Empirical Study: Objectives (2/3) Assess user´s perceptions of the usefulness and humanization of the system • Useful: helpful, convenient, comfortable. • Humanized: pleasant, unobtrusive, satisfying, reliable, efficient, likeable. PhD Examination

16. Empirical Study: Objectives (3/3) • Assess differences in those user’s perceptions according to psychological (MBTI) and other personal characteristics: • Heuristic (N, F) versus Analytic (S, T) psychological types • Intuition (N) / Sensing (S) dimension => preferences for info. Acquisition (awareness - first step in the acquisition of info.) • Feeling (F) / Thinking (T) dimension => preferences for info. processing • Gender, O.S. proficiency, familiarity with TTS PhD Examination

17. Empirical Study: Evaluation Method • Quasi-experimental model • Extensive laboratory test that simulated closely a real world situation. • 14 specific research questions • IV: subject variables; DV: subjective user’s opinions and perceptions • Participants • 15 participants selected from the population of the C.S. Department • Measurement of variables • Attitude measurement through the use of questionnaires • 5-point scales: 1=Strongly Disagree, 3=Neither disagree nor agree, 5=Strongly agree • 7-point scales: 1=Not at all, 4= Undecided, 7= Very much so • 7-point semantic differential scales: e.g. 1 = unpleasant, 7 = pleasant PhD Examination

18. Empirical Study: Experimental Design and Procedure PhD Examination

19. Empirical Study: Summary of Results (1/2) • Speech is useful to convey information that is hidden from view (but content must be humanized through context cues) • The prototype was perceived as supporting the awareness of visually hidden information. • It was considered particularly useful by heuristic information gatherers – speech-based alerts allowed participants to appreciate the immediate relevance of an informational event, that is the relationship and connection between the alerting information and their personal interests and activities. • Information awareness is an arena where using speech is a valid and beneficial approach • Participants indicated that speech-based interruptions were not overly disruptive and allowed them to decide when to deal with information. PhD Examination

20. Empirical Study: Summary of Results (2/2) • The anthropomorphic behaviours were considered to be important and adequate to reinforce the social acceptability of the speech-based system • Attentive interurptions, linguistic variation and politeness combine to avoid socially incorrect behaviours, such as respectively unnecessary interruptions, irritating repetitions and rudeness. • Urgency and priority-based interruptions were perceived as relevant. • Attention-grabbing and audio modulation were perceived as natural (or expected) behaviours. • When messages were varied they did not irritate participants. They were found so natural that some people did not even notice the variation. • Polite behaviour is appropriate to avoid rudeness when getting a person’s attention, but it invokes the perception of a personality and thus must be used in a way so that it suggests a consistent personality. PhD Examination

21. End of block 3 Questions Discussion

22. Summary of the Research Achievements • How to reinforce the social acceptability of a speaking system Humanize interruptions, vary language, use politeness • How to enhance information awareness Support speech-based alerts including contextual cues • Experimental environment: design and prototype Agent-based approach, protocol, delivery of spoken alerts • Empirical study Validated the anthropomorphic speech-based approach for info. awareness • Practical guidelines for the design of speech-based systems for information awareness. PhD Examination

23. Contributions (1/4) • The thesis contributes to understand the importance of, and how to exploit, speech to enhance the awareness of personal information that is hidden from view in a desktop-based environment: • How to humanize the delivery of speech-based alerts • How to humanize the contents of speech-based alerts PhD Examination

24. Contributions (2/4) • The thesis contributes with a unique experimental environment that integrates three different anthropomorphic characteristics for the delivery • Attentive interruptions, linguistic variation, politeness • They reinforce the social acceptability of the system - it was unexpectedly well accepted by users. PhD Examination

25. Contributions (3/4) • The thesis contributes to the research community in that it provides an extensible agent-based architecture for an experimental environment: • Can be extended with additional specialised agents • Can accommodate the incorporation of additional anthropomorphic behaviours (e.g. gender, personality, emotion) • Can be used to support controlled experiments with larger and more heterogeneous user groups PhD Examination

26. Contributions (4/4) • The thesis contributes to expand current knowledge about how to evaluate user’s perceptions of anthropomorphic systems that employ speech, and suggests a range of appropriate subjective measures • It measured the adequacy of speech-induced anthropomorphic behaviours through specific questionnaires • It measured the perceptions of usefulness and humanization through careful designed questionnaires PhD Examination