1 / 33

Real-Time Video for Computer Entertainment

Real-Time Video for Computer Entertainment

sldsrv
Télécharger la présentation

Real-Time Video for Computer Entertainment

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Real-Time Video for Computer Entertainment Richard Marks Sony Computer Entertainment America Research & Development Division

  2. Outline • • Background Background • • Video Input for PS2 Video Input for PS2 • • Natural Interfaces Natural Interfaces • • Conclusions Conclusions • • Future Work Future Work

  3. Sony Computer Entertainment • • Subsidiary of Sony Subsidiary of Sony • • Responsible for PlayStation Responsible for PlayStation- -related products products • • SCEI (Tokyo) SCEI (Tokyo) • • SCEE (London) SCEE (London) • • SCEA (Foster City, CA SCEA (Foster City, CA-- --near related near SanFran SanFran) )

  4. Sony Computer Entertainment America (SCEA) • • R & D: 15 people in Foster City, CA R & D: 15 people in Foster City, CA • • Mission: Catalyze new ideas for Mission: Catalyze new ideas for computer entertainment computer entertainment • • Focus: Software for PlayStation2 Focus: Software for PlayStation2 • Advanced rendering • Intelligent characters • Physical simulation • Digital interfaces

  5. My Background • • MIT, B.S. 1990 in Avionics MIT, B.S. 1990 in Avionics • • Stanford, Stanford, Ph Ph.D. 1995 in underwater .D. 1995 in underwater robotics robotics • • Teleos Teleos Research, 10 person computer Research, 10 person computer vision startup vision startup • • Autodesk Autodesk, makers of AutoCAD, worked , makers of AutoCAD, worked on photo on photo- -to to- -model research model research • • Digital Video Art, graphics consulting Digital Video Art, graphics consulting

  6. My Research Goals • • Explore new capabilities generated by Explore new capabilities generated by real real- -time video input to PS2 time video input to PS2 • Investigate alternative user-input mechanisms for PS2 (besides joystick) • Understand limitations of low-cost video input • Create new video-based entertainment models

  7. Real-time Video Research • • Natural Interfaces Natural Interfaces • SIGGRAPH 2000 sketch, emerging technology • Game Developer Conference 2001 • • Enhanced Reality Enhanced Reality • ACM1 • SIGGRAPH 2001

  8. PlayStation2, not PC • • Platform is constant Platform is constant • iLINK (IEEE1394) and 2 USB ports • Known compute capability (much) • • Very diverse audience Very diverse audience • • Unique architecture Unique architecture • Highly parallel • Micro-programmable • Data-centric

  9. PlayStation2 EE GS Emotion Engine Graphics Synthesizer RAM (32 MB) MIPS-based core 2 FP Vector Units Image Proc. Unit VIDEO OUT 4 MB on-die VRAM RAM (2 MB) SPU2 IOP AUDIO OUT 2 USB I/O Processor Sound Processor 1 iLINK (IEEE 1394) HD/ETHERNET ROM DVD/CD

  10. Current Setup • • USB USB webcam webcam (<$50) • 30 Hz YUV420 video • 320x240 compressed,160x120 uncompressed (<$50) • • Video processing performed by core Video processing performed by core • Decompression (bit-stream decode, IDCT) • Low-level image filters (smooth, threshold, etc.) • Segmentation, matching, tracking • • Demo Demo

  11. Natural Interfaces • • Intuitive Intuitive • • Simple Simple • • Enabling Enabling • • Enjoyable Enjoyable ⇒ ⇒ ⇒ ⇒ Video Video- -based interfaces, with and based interfaces, with and without props without props

  12. Specification • • Fun Fun • • Intuitive Intuitive • • Enabling Enabling • • Real Real- -time time • 30 frames/second • Less than 3 frames total latency • • Robust Robust • Graceful failure/error recovery

  13. Medieval Chamber • • Multiple color Multiple color- -based tracking approaches based tracking approaches • Richard Marks • • Advanced rendering including shadows, Advanced rendering including shadows, transparency, reflections, etc. transparency, reflections, etc. • Gabor Nagy • • Physical simulation/collisions Physical simulation/collisions • Eric Larsen

  14. Medieval Chamber • • Known camera, objects Known camera, objects • Spheres and cylinders have special projection properties • • Tracking steps Tracking steps • Color segmentation • Centroid, moment calculation • Windowed centroid, moment calculation • Color-transition detection • Situational probabilistic ambiguity resolution • Kalman filtering

  15. Medieval Chamber • Sphere – x,y from centroid, – z from principal moment – Rx, Ry from dot centroid (given x,y) • Cylinder – x,y from centroid – Rz from angle of principal moment » Marker used to resolve ambiguity – z from secondary moment – Body Rx from principal moment (given z) » Foreshortening used to resolve ambiguity – Body Ry from helical stripe Y X Y X

  16. Medieval Chamber • • Combination of sphere and cylinder Combination of sphere and cylinder provides most robust tracking provides most robust tracking

  17. Marionette • • Alternative form of character control Alternative form of character control • • Traditional marionette Traditional marionette • Darwin the Wizard, created by Daniel Oates • • Virtual marionette Virtual marionette • 3D model by Care Michaud

  18. Marionette • • Color segmentation Color segmentation • • Line fitting to find T shape Line fitting to find T shape • • T shape analysis to recover puppet T shape analysis to recover puppet parameters parameters

  19. Planet Explorer • • 3D viewing, navigation 3D viewing, navigation • • Earth rendering Earth rendering • Greg Corson • • Rotating the ball rotates the earth Rotating the ball rotates the earth • • Proximity of ball to camera adjusts zoom Proximity of ball to camera adjusts zoom

  20. Planet Explorer • • Color segmentation/ Color segmentation/centroid • • Principal moment to adjust zoom Principal moment to adjust zoom • • Motion Motion- -estimation to measure rotations estimation to measure rotations centroid to find ball to find ball

  21. Fly • • Flight simulation Flight simulation • • Procedural landscape Procedural landscape • Tyler Daniel • • Relative arm angles determine bank angle Relative arm angles determine bank angle • • Average arm angles determine attack angle Average arm angles determine attack angle • • Arm motion increases airspeed Arm motion increases airspeed • • Shadow wings mimic arms Shadow wings mimic arms

  22. Fly • • Centroid Centroid/ moment determines body extent / moment determines body extent • • Principal axis angles of outer regions Principal axis angles of outer regions correspond to arm angles correspond to arm angles

  23. Conclusions • • Known props can provide more Known props can provide more information and still be natural information and still be natural • • Area Area- -based measurement more robust based measurement more robust and precise than lineal measures and precise than lineal measures • • Precision more important than accuracy Precision more important than accuracy • • Color sensitive to lighting, but color Color sensitive to lighting, but color transitions are not transitions are not

  24. Conclusions (contd.) • • Make signal proportional to action Make signal proportional to action • • Secondary motion can enhance Secondary motion can enhance perceived response (and hide error) perceived response (and hide error) • • Display perspective important Display perspective important

  25. Future Work • • Attach interfaces to Attach interfaces to real • • Investigate interfaces for creative Investigate interfaces for creative content generation content generation • • Enhanced Reality research Enhanced Reality research real games games

  26. Enhanced Reality: What is it? • • Starting with live audio/video, enhance it Starting with live audio/video, enhance it by adding/modifying the content by adding/modifying the content • • Not virtual reality! Not virtual reality! • • Similar to Augmented Reality (a la MIT Similar to Augmented Reality (a la MIT Media Lab), but with entertainment focus Media Lab), but with entertainment focus • • Consists of both enhancements to user Consists of both enhancements to user and enhancements to environment and enhancements to environment

  27. Virtual Pet

  28. Color Transitions • • Project ( Project (Cr pixel onto a line pixel onto a line • • Similar to barcodes, Similar to barcodes, but selectable but selectable • • Maximal separation Maximal separation produces best results produces best results • • Robust to lighting Robust to lighting variation variation • • Patents pending Patents pending Cr, , Cb Cb) for each ) for each Cb Green Cr Magenta

  29. Hand Puppet • • Very simple form of character control Very simple form of character control • • 3D models by Guy 3D models by Guy Burdick Burdick

  30. Hand Puppet • • Color segmentation to locate hands Color segmentation to locate hands • • Split screen for left/right hands Split screen for left/right hands • • Centroid Centroid, moments and principal moment , moments and principal moment orientation orientation • • Split along principal moment, principal Split along principal moment, principal moment orientations provide mouth angle moment orientations provide mouth angle

More Related