Cpre 185 Wii/Kinect Lab Software

Cpre 185 Wii/Kinect Lab Software Senior Design Group 16 Client / Adviser: Dr. Tom Daniels BrentonHankins, Rick Hanton, Harsh Goel, Jeff Kramer

Problem Statement • Freshmen computer engineering college students and high school students need an interesting way to be introduced to programming and problem solving. • In the past, Iowa State students have been taught to create simple programs in the C programming language that use Nintendo Wiimotes as input devices. • Software needs to be developed similar to Wii Wrap, but uses the Microsoft Kinect™ sensor as input to student-written programs.

Hardware - Kinect • RGB Webcam Camera 640x480 @ 30 Hz (center) • IR Projector Projects Grid for Depth Mapping • IR Camera at 640x480 @ 30 Hz collects depth data (right) • Microphones give room audio • Small Panning motor turns sensor bar from side to side • Onboard image preprocessing • Projector power draw requires extra AC power

Hardware - Wii • Wii Remote / Wiimote • 11 buttons + Power • Roll/Pitch/Yaw Gyro • Infrared Sensor • Speaker (unsupported) • Optional Nunchuck

Schedule / Cost

Architecture • Three Modules: • Hardware interface to read in raw data (provided) • Interface simplification / data processing • Graphical interface to display output

Flow Diagram

WiiMote Solutions • Issue: Decrease delay in printf output • Solution: Added a fflush(stdout) statement immediately after printf • Issue: IR data becomes erratic if sensor loses source for one frame of data • Solution: Save last good frame of IR data (x,y) and loop the last good data until new data arrives

Kinect Design Approaches • OpenKinect • Tricky to get started with in windows • Natively supports Unix • Body tracking not built-in • Needs a calibration pose for body tracking • No event when new Video or Depth frame is available • Microsoft Kinect SDK • Sample Projects • Documentation • Simple installer • Natively supports Windows • Full body tracking built-in

Microsoft Kinect SDK • Data Available • RGB Video Frames(640x480 max resolution) • Depth (mm) Data Frames (320x240 max resolution) • Skeletal tracking points (20 points per player) • How we get the data • Pass an event object to alert us when new data is available • Pass a video/depth/skeleton frame pointer to read in next frame • Run event loop and process each frame as RGB/Depth/Skeleton data

Shared Memory Use • Windows File Mapping is an inter-process communication tool that maps a block of memory by name • Shared Memory in Kinect Wrap • File Mapping is set up by Kinect Wrap backend program with a specific name for each block • Backend gets data from Kinect SDK and dumps it into shared memory space when each new frame becomes available • Our API Library copies frame from shared memory into local memory space for student program to use.

Kinect Wrap API • Provides an easy-to-use interface for students while maximizing functionality • Functions: • getVideoFrame(rgb** pFrame) • getDepthFrame(short** pFrame) • getSkeleton(Point3D* pSkel) • setOutputDisplay(rgb** pFrame) • setOutputText(char* sText) • setPixel(rgb** pFrame, int x, int y, KCOLOR color)

Kinect Wrap Library • Kinect Wrap Library is a static Windows library that students must include their C projects • Library handles backend processing of API • Abstracts all of the technical workings of the Kinect SDK, File Mapping API, and GUI design

Kinect GUI • Abstracts away the details of drawing Kinect data via a graphical API • Kinect GUI receives data via shared memory • Data is drawn as a grid of vertices, each with a custom RGB value via DirectX • Intend to port GUI code to OpenGL and work on constructing a more intuitive GUI in future

Testing – Wiimotes • Field Testing • Compared time delay of data transmission between the current and previous iterations of the Wii Wrap code. • Found the apparent delay seen in the Wii Wrap GUI to be significantly decreased. • Test results corroborated by CPRE 185 lab TAs during classroom use. • No previous functionality was affected by these changes.

Kinect Black Box Testing • Assumed that the Kinect SDK functions deliver correct data from Kinect sensor • Shared Memory • Wrote known data into shared memory space with one program and had second program read it out and verify that the data has not been altered • Student API Interface • Verified that buffers of known data sent to the student program are not altered before being read • Verified that a known function run by student code produces the expected output frame data given a known input frame

Kinect Black Box Testing • GUI Output • Tested whether frames were being updated consistently by writing specific colors to known pixels in a given output frame

Kinect Integration Testing • setPixel() • Tested for boundary values. • Attempted to set pixels outside the frame (640x480) • Attempted to set pixel RGB values outside of 0-255 • getVideoFrame() & setOutputDisplay() • Check that output frame RGB data matches input data • Found that with simple student processing code, any reduction in output fps from device frame rate is minimal • getDepthFrame() • Used a plane at a known distance from the Kinect to find how accurate the depth values in mm are vs. reality.

Kinect Lab Example intmain() { KinectLib_Init(); // Initialize the library rgb* frame[VIDEO_HEIGHT]; // Create 2D frame of rgb data short* dframe[DEPTH_HEIGHT]; // Create 2D frame of short depth data Point3D* skeleton; // Create array of skeleton data while(1){ getVideoFrame(frame); // Map video frame to pointers getDepthFrame(dframe); // Map depth frame to pointers if(skeletonTracked()) { skeleton = getSkeleton(); // Map Skeleton data } setPixel(frame, 23, 18, BLUE); // Arbitrarily set pixel (23, 18) to Blue displayVideoFrame(frame); // Display the modified video frame in the GUI } KinectLib_Destroy(); // Destroy Kinect library instance return 1; }

Future Work • Multi-Kinect Support • More comprehensive GUI program • Additional Library Functionality • Create code to share Kinect data over Ethernet

Questions? seniord.ece.iastate.edu/dec1116/

Cpre 185 Wii/Kinect Lab Software