Sign Language Recognition

1. Byron Hood | version 0.3 Computer Systems Lab Project 2007-2008 Sign Language Recognition

2. Byron Hood | version 0.3 Computer Systems Lab Project 2007-2008 Overview • Average person typing speed • Composing: 19 words per minute • Transcribing: 33 words per minute • Sign speaker • Full sign language: 200—220 wpm • Spelling out: est. 50-55 wpm • Faster by 1.5x to 3x

3. Byron Hood | version 0.3 Computer Systems Lab Project 2007-2008 Purpose and Scope • Native signers can input faster • Benefits: • Hearing & speaking disabled • Sign interpreters • Just letters & numbers for now

4. Byron Hood | version 0.3 Computer Systems Lab Project 2007-2008 Research • Related projects • Using mechanical gloves, colored gloves • Tracking body parts • Neural network-based application • Still images: 92% accuracy • Motion: less than 50% accuracy • Feature vector-based application • Also about 90% accuracy on stills • No motion tests

5. Byron Hood | version 0.3 Computer Systems Lab Project 2007-2008 More Research • Image techniques: • Edge detection (Robert’s Cross)‏ • Line detection (Hough transform)‏ • Line interpretation methods • Chaining groups of lines • Macro-scale templates

6. Byron Hood | version 0.3 Computer Systems Lab Project 2007-2008 Program Architecture • Webcam interface • Edge detection • Line detection • Line interpretation • Hand attribute matching

7. Byron Hood | version 0.3 Computer Systems Lab Project 2007-2008 Testing Model • Human interaction necessary • General testing model: bhood@bulusan:~/syslab-tech$ \ > ./main images/hand.pgm in.pgm [DEBUG] Edge detect time: 486 ms [DEBUG] Wrote image file to `in.pgm’ Errors:0Warnings:0

8. Byron Hood | version 0.3 Computer Systems Lab Project 2007-2008 Code sample • Robert’s Cross edge detection: int row,col; // loop through the image for(row=0; row<rows; row++) { for(col=0; col<cols; col++) { // avoid the problems from trying to calculate on an edge if(row==0 || row==rows-1 || col==0 || col==cols-1) { image2[row][col]=0; // so set value to 0 } else { int left = image1[row][col-1]; // some variables int right = image1[row][col+1]; // to make the final int top = image1[row-1][col]; // equation seem a int bottom = image1[row+1][col]; // little bit neater image2[row][col]=(int)(sqrt(pow(left – right , 2) + pow(top - bottom, 2))); } } }

9. Byron Hood | version 0.3 Computer Systems Lab Project 2007-2008 Edge Detection Results Original image (800 x 703)‏ • Results: • Shows the important edges but little else • Robert’s Cross method the optimal balance Final image (800 x 703)‏

10. Byron Hood | version 0.3 Computer Systems Lab Project 2007-2008 Cropping Results • Remove useless rows with no features • Very large optimization • Area difference • Input/output Original (800 x 703)‏ Result (633 x 645)‏

11. Byron Hood | version 0.3 Computer Systems Lab Project 2007-2008 Line detection • Mostly complete • Still some bugs to iron out • Shows nonexistent lines occasionally (see image!)‏

12. Byron Hood | version 0.3 Computer Systems Lab Project 2007-2008 Line Interpretation • Chaining groups of lines • Templates • Generation • Template-based comparison • Hand matching

13. Byron Hood | version 0.3 Computer Systems Lab Project 2007-2008 The Mysterious Future • Finish line interpretation • Building method for interpreting lines • Working on template-based recognition • Write XML template files • Finish camera-computer interaction • Device control must be precise, picky