SceneMaker: Multimodal Visualisation of Natural Language Film Scripts

1. SceneMaker:Multimodal Visualisationof Natural Language Film Scripts Dr. Minhua Eunice Ma Eva Hanser Prof. Paul Mc Kevitt Dr. Tom Lunney Dr. Joan Condell School of Computing & Intelligent Systems Faculty of Computing & Engineering University of Ulster, Magee, Northern Ireland hanser-e@email.ulster.ac.uk, {p.mckevitt, tf.lunney, j.condell}@ulster.ac.uk School of Computing and Mathematics Faculty of Business, Computing and Law University of Derby, England m.ma@derby.ac.uk

2. PRESENTATION OUTLINE Aims & Objectives Related Projects SceneMaker Design and Implementation Relation to Other Work Conclusion and Future Work

3. AIMS : AIMS & OBJECTIVES • Automatically generate well-designed and affective virtual scenes from screenplays • Realistic visualisation of emotional aspects • Multimodal representation with 3D animation, speech, audio and cinematography • Enhance believability of virtual actors and scene presentation Input: SceneMaker System Screen- play Output: Animation

4. OBJECTIVES : AIMS & OBJECTIVES • Processing/inferencing emotions and semantic information within story context • Common sense, affective and cinematic knowledge ontologies reflecting human cognitive reasoning rules • Automatic genre recognition from text • Design, implementation and evaluation of SceneMaker

5. SEMANTIC TEXT PROCESSING : RELATED PROJECTS INT. M.I.T. HALLWAY -- NIGHT Lambeau and Tom come around a corner. His P.O.V. reveals a figure in silhouette blazing through the proof on the chalkboard. There is a mop and a bucket beside him. As Lambeau draws closer, reveal that the figure is Will, in his janitor's uniform. There is a look of intense concentration in his eyes. LAMBEAU Excuse me! WILL Oh, I'm sorry. LAMBEAU What're you doing? WILL (walking away) I'm sorry. • Standardized format and language of screenplays • Automatic annotation of formal screenplay elements (Jhala 2008) • Semantic information on location, timing, props, actors, events, manners, dialogue and camera direction Screenplay Extract from ‘Good Will Hunting (1997)’

6. VISUAL AND EMOTIONAL SCRIPTING : RELATED PROJECTS • Emotion recognition from text:keyword spotting, lexical affinity, statistical models, fuzzy logic rules, machine learning, commonsense knowledge, cognitive models • XML-based annotations defining visual appearance of animated characters and scenes:BEAT – Behaviour Expression Animation Toolkit (Cassell et al. 2001)MSML – Movie Script Markup Language(Van Rijsselbergen et al. 2009) <GAZE word=1 time=0.0 spec=AWAY_FROM_HEARER> <GAZE word=3 time=0.517 spec=TOWARDS_HEARER> <R_GESTURE_START word=3 time=0.517 spec=BEAT> <EYEBROWS_START word=3 time=0.517>

7. MODELLING AFFECTIVE BEHAVIOUR : RELATED PROJECTS • Automatic physical transformation and synchronisation of 3D models reflecting emotion • Manner influences intensity, scale, force, fluency and timing of an action • Multimodal annotated affective video or motion captured data (Gunes and Piccardi 2006) Greta (Pelachaud 2005) Personality & Emotion Engine(Su et al. 2007)

8. VISUALISING 3D SCENES : RELATED PROJECTS • WordsEye – Scene composition(Coyne and Sproat 2001) • ScriptViz – Screenplay visualisation(Liu and Leung 2006) • CONFUCIUS – Action, speech & scene animation(Ma 2006) • CAMEO – Cinematic and genre visualisation(Shim and Kang 2008) ScriptViz CONFUCIUS CAMEO WordsEye

9. AUDIO GENERATION : RELATED PROJECTS • Emotional speech synthesis (Schröder 2001) - Prosody rules • Music recommendation systems - Categorisation of rhythm, chords, tempo, melody, loudness and tonality - Sad or happy music and genre membership (Cano et al. 2005) - Associations between emotions and music (Kuo et al. 2005)

10. KEYOBJECTIVES : DESIGN AND IMPLEMENTATION • Context consideration through natural language processing, common sense knowledge and reasoning methods • Extract genre and moods from screenplays • Influence on all elements of visualisation • Enhance naturalism and believability • Text-to-animation software prototype, SceneMaker

11. Screen-play Genre Multimedia Generation Text & Language Processing } } Script Editor Emotion Animation Player Context Interpretation Action ARCHITECTURE OF SCENEMAKER : DESIGN AND IMPLEMENTATION

12. Natural Language Processing & Script Segmentation Context + Emotion Reasoning Event Synchronisation 3D Rendering + Multimedia Gate(1) ANNIE Onto-Gazetteer ConceptNet(3) Common Sense Knowledge • MSML(5) • /SMIL Unity(6) 3D Engine (JavaScript,XML) Movie Script GenreOntology RDFS/OWL Movie Ontology RDFS/OWL WordNet-Affect(4) Automatic Sound & Music Selection 3D Models (3D Studio Max) Festival(7) Speech Synthesiser LVSR(2) Lexical Visual Semantic Representation Script FormatOntology SOFTWARE AND TOOLS : DESIGN AND IMPLEMENTATION (1) http://gate.ac.uk (2) Ma 2006 (3) Liu and Singh 2004 (4) Strapparavaand Valitutti 2004 (5) Van Rijsselbergen et al. 2009 (6) http://unity3d.com (7) http://www.cstr.ed.ac.uk/projects/festival

13. EVALUATION OF SCENEMAKER : DESIGN AND IMPLEMENTATION Evaluating 4 aspects of SceneMaker:

14. RELATION TO OTHER WORK

15. CONCLUSION AND FUTURE WORK • Automatic expressive multi-media animation of screenplays • Focus on:– automaticreasoning about story context and emotional interpretation – based on world knowledge and context memory – emotions influencing scene compositions and event execution– scene direction refined by genre-specifics • Analysis of script format to access semantic information • Automatic genre specification from script • Heightened expressiveness, naturalness and artistic quality • Assist directors, actors, drama students, script writers • Future work: Implementation & Testing of SceneMaker

16. Thank you.QUESTIONS OR COMMENTS?