Digital Interactive Entertainment

1. Digital Interactive Entertainment Dr. Yangsheng Wang Professor of Institute of Automation Chinese Academy of Sciences Yangsheng.wang@mail.ia.ac.cn http://hci.ia.ac.cn/JointLab/onlinegame-En.htm

2. Overview ofDigital Entertainment Industry

3. Digital Entertainment Industry • Year 2008 the market size will be about $100B and major products are TV games、PC games and mobile games. • Microsoft, Sony and Nintendo provide most digital entertainment products. • China has a very big potential market for digital entertainment products.

4. Internet and Game in China • The total amount of Chinese Internet users is up to 87 millions in 2004, about 25% are online game(OG) users. • OGsareone of major parts of digital entertainment contents in China.Almost half of the OGs on the domestic market mainly come from Korea. Some Chinese game companiers are beginning to develop their own OGs. • PC Stand-alone games are losing their market. • Mobile games are developed rapidly, but the income is on the low level at the present stage. • Overseas game companies enter Chinese market via Joint-venture.

5. Internet Users in China source：CGPA&IDC 2004

6. Online Game Market in China source：CGPA&IDC 2004

7. Stand-alone Game Market in China source：CGPA&IDC 2004

8. New Research & Applications ofNatural Interactive Technologies

9. Natural Interactive Technologies (NIT) • Traditional games mainly depend on mouse, keyboard, joystick and special input device. • Game players desire more natural and intelligent interaction methods, including voice, gesture, expression, gaze, and intelligent sensor, etc. • User-oriented multi-model HCI (Human Computer Interaction) technologies has become the hotspot of research and application in many domains.

10. NIT Research and Application-I • Computer Vision for Computer Games and Graphics • Camera is used as a powerful interface device for games by sensing user’s body position, head orientation, direction of gaze, pointing commands, and gestures. • Developer: Mitsubishi Electric Research Laboratories. • Applications: Sega games Vision device

11. NIT Research and Application-II • Vision-based Human-Computer Interface for Games • By a camera, the player's body movements are projected right into the game as an unencumbered input and control device. • Developer: Sony Computer Entertainment America • Applications: Sony PlayStation2 EyeToy games Camera

12. NIT Research and Application-III • Voice Control for Games and Simulations • Many of today's games have a lot of keyboard commands that can be hard to remember. Speech recognition technologies are used in this area for game control. • Product: Game Commander (Mindmaker) • Product: Game Voice (Microsoft SideWinder)

13. Interactive Entertainment Research Group in CASIA

14. The Outline of Our Research • Auto-Construction of a Game Object; • Player-Oriented Game • Multi-model Human & Computer Interaction; • Natural Interactive Entertainment Platform.

15. Photo-stereo Based Model Reconstruction flashlight • Images capture • Normal calculation • Surface reconstruction bracket camera flashlight flashlight control card

16. Images capture Images of the surface under different lighting conditions is captured in a very short period, that the register is avoid.

17. Normal calculation A photo-stereo method is used to retrieve the surface normal based on the reflect property of the surface.

18. Surface reconstruction • The relevant height between the points can be integrated from the derivative field. • A pyramid integration method is used to optimize the results

19. Orthogonal images Based Model Reconstruction • Generic Model and its feature pointsselection • Two images feature points editing • Generic model deformation • Texture merging and mapping

20. Generic model and its feature points selection 79 feature points according MPEG4

21. Images feature points editing andtexture merging frontal face side face texture merging result

22. Generic model deformation andtexture mapping texture generic model deforming model reconstruction result

23. Individual game characters player A Individual faces give players more fun and immersion. player B

24. Video-Based Model Reconstruction and Animation Capture user’s frontal face with a camera, then deform an existent face model and map the face image to create an individual face model. Five kind of user’s expressions are recognized real-time to drive the relevant animations. • Face detection • Deformation & Texture mapping • Expression recognition • Expression retargeting & Animation

25. Auto-Extraction of key points of a face An RealBoost-based statistical learning is adopted to build a model for each facial point. During extraction, the models are used to locate facial points for each face image.

26. Deformation & Texture mapping generic model reconstructed individual models deforming Origin image filling image merging texture

27. Expression recognition and retargeting A novel approach to real time facial expression recognition using an embedded HMM which uses observation vectors that are composed of DCT results. Source expression models Expression recognition Individual expression results surprise anger happy fear sad

28. Getting into a Game -Change face • Player’s frontal face is captured by a camera. • Player’s face feature points are detected to deform character’s face model. • Player’s face image is used change character’s face texture.

29. Video-Based Motion Detection and Analysis • Head detection and pose estimation • Fist detection and motion analysis • Other motion detection and analysis

30. Head detection and pose estimation front left right up down close far The novel scheme based on Markov model fusing appearance, color, and motion information. • locating the position of a head • recognizing the pose of the head

31. Fist detection and motion analysis training samples prototype DEMO

32. Driving a car using your hands and voice • Player’s hands control the car’s direction. • Player’s voice controls the car’s speed and status.

33. Motion Detection and Analysis • Project video into game • Motion detection • Parameters calculation • Game objects control

34. Playing a game with your body • Player’s body movement is projected right into a game • The motion detection and analysis results are used to activate objects in a game.

35. Online-Game Strategic Research Group

36. Online-Game Strategic Research Group The group includes major digital entertainment companies, universities, research institutes, and digital entertainment association in China. Ameeting of the group

37. The goal of the Strategic Research Group • Write strategic report of digital entertainment for Chinese government. • Establish “China National Center for Digital Entertainment”.

38. Thank you!