ATLAS : Automatic Translation into Sign Languages

1. ATLAS : Automatic Translation into Sign Languages Paolo PRINETTO Politecnico di Torino (Italy)University of Illinois at Chicago, IL (USA) Paolo.Prinetto@polito.itprinetto@uic.edu www.testgroup.polito.itwww.comitato-girotondo.org

2. Expected key questions … • What are Sign Languages? • Why are you going to talk about them? • Why are you going to talk about them here at EWD&T 2009? • …

3. What are Sign Languages?

4. Sign Languages • Sign Languages are real languages like any other human languages and not just “pictures in the air”. • The European Parliament, on June 17th, 1988 and November 18th, 1998, and the UNESCO in Salamanca Resolution, art. 21, on May 15th, 2001, stated the official status of Sign Languages of Deaf people as official languages.

5. Sign Language is not Universal • Lingua dei segni italiana (LIS) • American Sign Language (ASL) • British Sign Language (BSL) • Langue des Signes Française (LSF) • Lingua dei segni nicaraguense (ISN) • New Zealand Sign Language (NZSL) • …

6. Deaf people in Italy • In Italy 1 person out of 1,000 is affected by serious deafness • In Italy there are today about 50,000 persons who were born deaf or got deafness in the first years of life

7. Impacts • The integration difficulties of persons who were born deaf or got deafness in the first years of life are higher because they could not acquire the knowledge of the spoken language. • Sign Languages allow deaf children acquiring a full cognitive development within their community composed by hearing and deaf persons. • Such cognitive development represents the prerequisite to a full access to the education, the culture and the inclusion in working and social environment.

8. Why am I going to talk about them? • I’m chairing and managing the ATLAS Project: • a 3-year project, started January 2009 • co-funded by Regione Piemonte, Piedmont Region, Turin, Italy • www.atlas.polito.it

9. The ATLAS Project • The ATLAS project exploits the convergence between cognitive sciences and ICT to build innovative services to provide deaf people with the access to broadcast communi-cations, through the automatic translation of the written Italian natural language into the animated Italian Sign Language (LIS)

10. The ATLAS Project (3) • Such a challenging goal will be achieved by pursuing the following objectives: • Develop an automatic translation system from Italian language written text (subtitles or general texts) to LIS. The translation will be accomplished resorting to the most advanced statistic and rule-based techniques • Develop appropriate “corpora” of information to properly train the automatic translation system.

11. The ATLAS Project (4) • Develop a visual assisted editor to support interpreters in the manual translation from Italian texts to ISL. The editor will serve as the reference tool for building the corpora • Develop an engine for rendering and visualization of a virtual interpreter

16. The ATLAS Project (5) • Define avatar commands for different delivering platforms and client devices • Design and deploy of client server platforms to demonstrate the service on different media, including • several types of Digital Video Broadcasting (DVB) • mobile phones • web applications • additional physical media such as DVD.

17. ATLAS Architecture Partial Overview

19. ATLAS Architecture Partial Overview

26. ATLAS Architecture Partial Overview

27. The challenge • Representing sentences in Sign Languages resorting to the minimum set of information needed to drive the virtual character.

28. Open issues in ATLAS_Written_ISL • Sign spatial positions (i.e., 3-dimensional starting location & ending locations) are context sensitive • Sign classifications: • Fixed position • “Movable” • Placeholders are extensively used

29. “The car is parked between the cat and the house” • Placeholder for car, house, cat

30. An (incomplete) example • Input: • La mamma ha dato al bambino un grosso gelato alla pesca. (The mother gave the kid a big peach ice cream). • Output: • MAMMA(1) • BAMBINO(2) • GELATO(3; scaled_of=big) • PESCA$1(4; Attr_of=3) • DARE(src=1; dst=2; obj=3; temp_attr=done)

31. Sign Representation At least 5 parameters: Movement Hand Shape Location Orientation Non-Manual Gesture Signs (NMGSs) or facial expressions are required to describe signs properly.

32. ATLAS Architecture Partial Overview

33. 3D (Character) Animation • Keyframe interpolation • Manually edited static poses (keyframes) • Algorithmic interpolation • Motion Capture • Animation data curves recorded from the real world • Physical simulation • Weight and forces to body parts • Purely algorithmic

34. Keyframe animation

36. Geometries

37. Motion Capture Recording real human or (cooperative) animal motion

38. Motion capture

39. Screen Translation Chain Simplified Italian text NL → EWISL Translator EWISL text EWISL → AL Translator Animation Language text “Actionary” [Badler00] Action/Animation Dictionary. A map between words and actor movement. Character 3D Model AL text structure: AL Player Renderer <time> <action> <params ...> <time> <action> <params ...> <time> <action> <params ...> <time> <action> <params ...> <time> <action> <params ...> ... Sequencer (Parse the timeline to issue commands at the right moment) Ogre & C. Broadcast point

40. Delivery Architecture Delivering center archivio Automatic Translation Assisted Translation pc hh stb ip ip dvb [ip] User terminals

41. A very first result

43. Cпаcибo!!! Малые Автюхи, Калинковичский район, Республики Беларусь