ON-LINE HANDWRITING RECOGNITION OF MEETING NOTES USING PEN PROSODY

1. ON-LINE HANDWRITING RECOGNITION OF MEETING NOTES USING PEN PROSODY Develop a novel way to improve on-line character recognition performance of handwritten notes using hidden Markov models with multi-space probability distributions.

2. Off-line Handwriting Recognition

3. On-line Handwriting Recognition

4. Off-line/On-line Handwriting Recognition

5. Handwriting Recognition • Machine Learning • Neural Networks • Connectionist Computing • Natural Language Processing • Signature Verification • Postal-Address Interpretation • Bank-Check Processing

6. Ami Corpus Augmented Multi-Party Interaction http://corpus.amiproject.org/ “A 15-member multi-disciplinary consortium dedicated to the research and development of technology that will help groups interact better.”

8. Pen Data • Digital pens and whiteboards are used by participants to make hand written notes. • Time stamped x-y co-ordinates of pen strokes are captured. • The pen strokes are stored as xml files for subsequent processing. • Prosody-like features such as stroke direction, duration and pen speed are captured.

9. Pen Data <Stroke> <StrokeIdx>0</StrokeIdx> <ID>0</ID> <StartDate>2004-12-13T10:47:45.0000000-00:00</StartDate> <StartMs>0</StartMs> <Duration>506</Duration> <Color>-16776961</Color> <Width>1</Width> <Points> <X>132.5</X> <Y>27</Y> </Points> . . . . <Points> <X>146.875</X> <Y>39.25</Y> </Points> <Forces>6E:72:72:72:68:4C:00:02:3C:5A:6E:74:70:6E:60:5A:52:5C:74:74:66:00:</Forces> </Stroke>

10. Pen Data • The XML files are converted into JPEG images (offline representation). • A DIV-X movie of these files is also produced (online representation). • A transcription of written data is also generated using an optical character recognition technique based on Liwicki and Bunke(2005).

11. Classification Each written character is represented by a vector sequence of observations O defined as: so then the probability of a word given a particular sequence can be computed using Bayes rule

12. Markov Models • Direct estimation of the joint conditional probability is not practicable. • Parametric model is required

13. Markov Model

14. Hidden Markov Models

15. Hidden Markov Models • Replace with the much simpler problem of estimating the Markov model parameters

16. Hidden Markov Models • HMM’s are categorized into discrete and continuous. • Handwriting can contain both. • Multi-space distribution HMM (MSD-HMM), which is an extended version of HMM, will be employed to address the problem.

17. The Hidden Markov Model Toolkit “Portable toolkit for building and manipulating hidden Markov models.” • Speech recognition • Speech synthesis • DNA sequencing • Character recognition

18. HTK Tools • Training • Testing • Analysis • Continuous density mixture Gaussians • Discrete distributions • Does not support MSD’s!!!

19. Next Steps • Learning to use the HTK. • Research into using multi space probability distributions in HMM’s. • Adapting HTK to utilise multi space probability distributions. • Testing and comparing accuracy with normal HMM (both discrete and continuous). • Write up paper and prepare final presentation.

20. Questions?

21. Sources • AMI Project Website - amiproject.org • A tutorial on Hidden Markov Models and selected applications in speech recognition -IEEE White Paper by L R Rabiner (1989) • HMM-Based On-Line Recognition of Handwritten Whiteboard Notes – M Liwicki, H Bunke • HTK Handbook • HTK Website - htk.eng.cam.ac.uk • Multi-Space Probability Distribution HMM - IEICE White Paper by K Tokuda, T Masuko, N Miyazaki and T Kobayashi (2000)