Ying Yin 1,2 , Tom Ouyang 1 , Kurt Partridge 1 , and Shumin Zhai 1

1. Making Touchscreen Keyboards Adaptive to Keys, Hand Postures, and Individuals– A Hierarchical Spatial Backoff Model Approach Ying Yin1,2, Tom Ouyang1, Kurt Partridge1, and Shumin Zhai1 1 Google Logo here 2MIT Logo here

2. Foundations to current methods • Language modeling • vocabulary • 1-gram, 2-gram … N gram frequencies • Spatial models • converting input touch points into probabilities of letters • Edit distance correction • assigning cost to insertion, deletion, and other spelling errors User and posture independent

3. Research questions • One promising area for improvement is by making them adapt to the user • What types of adaption are possible? • How do they affect performance?

4. Contributions • A novel hierarchical adaptive model • Show benefits of posture and user adaptation • Online posture classification method • 13.2% reduction in character error rate • compared to base model • without language model

5. Types of adaptation • Individual differences (cf. Findlater & Wobbrock, 2012) • Furthermore, people use different hand postures to type (cf. Azenkot and Zhai, 2012)

6. Different typing postures: two thumbs, one finger, or one thumb

7. Types of adaptation • Different postures  different touch patterns • Touch patterns also depend on letter keys Need adaptation (Azenkot & Zhai, 2012)

8. Challenges of adaptation • Complexity • three adaptive factors: key, posture, individual • large number of submodels • need sufficient data to build each submodel • Model selection • wrong selection may hurt keyboard quality • uncertainty in posture classification

9. Hierarchical spatial backoff model (SBM)

10. Hierarchical spatial backoff model (SBM) • Combinatorial and fine grained adaptation • Conservative • Does not require an extra training phase • Updates the model continuously online

11. Research method • “Pepper” dataset (Azenkot & Zhai, 2012) • 30 right-handed participants • given random phrases to type • between-subject: each person uses one posture • 84,292 touch points in total • 10-fold cross validation

12. Comparison of spatial models

13. Effective key areas Two-thumb One-finger

14. Posture classification • SVM-based classifier • Based on correlation between time and distance between consecutive touch points • no additional sensors required • speed independent • 86.4% accuracy • Real-time

15. Posture adaptation

16. Individual adaptation

17. Prototype implementation of SBM • 13.2% reduction in character error rate • compared to base model • without language model • Integrated with real keyboard • combined with language model • runs on Android phone in real-time

18. Future work • Weighted average of submodels instead of making binary decisions • More data: real-use logging and game playing • User studies • validate the accuracy and speed improvement • how users adapt their behavior to SBM • Combine spatial and language models

19. Contributions • A novel hierarchical adaptive model • Show benefits of posture and user adaptation • Online posture classification method • Opens up many more interesting HCI questions

20. Q & A

22. Prototype implementation of SBM • Posture & key adaptation models • supervised and batch learning • Individual adaptation models • unsupervised and online learning • Backs-off to more basic models when • posture estimation is uncertain (conf. < 0.94) • there is insufficient user data (< 50 data points)