触控与多点触控交互技术

1. 触控与多点触控交互技术 王锋 昆明理工大学 E-mail: wangfeng@acm.org Phone: 13700600260

2. 提纲 • 自然人机交互(NUI) • 触控技术的历史 • 触控研究的重点与目的 • 我们的工作

3. 人机交互 – UI • 用户界面(UI)技术是人机交互研究的热点问题之一。 • 一种新的人机交互设备的出现，都会引发用户界面技术的一次重要变化 • 用户界面在人机系统中负责计算机的输入和输出，产生必要的反馈，并直接影响最终用户对系统的使用。

4. What is HCI? • User：Am I error again? Computer for People? Or People for Computer???

5. Where are we going?

6. 董士海先生等曾提到过，用户界面: • 批处理 (BATCH PROCESSING) • 命令行 (CLI) • 图形用户 (GUI) • 自然用户界面(NUI)

7. 当前的人机交互研究正向更自然、更高效、更智能的方向，特别是朝着“用户自由”的方向发展。当前的人机交互研究正向更自然、更高效、更智能的方向，特别是朝着“用户自由”的方向发展。 • 用户逐渐成为交互关系中的主体，以个性化和场景自动感知的新的交互模式得到重视。 [1] J. Canny, "The Future of Human-Computer Interaction," Queue, vol. 4, pp. 24-32, 2006. [2] B. Myers, "A brief history of human-computer interaction technology," interactions, vol. 5, pp. 44-54, 1998.

8. Shneiderman指出，新的计算技术将更为关注人能做什么，但是人的天性和需要并不会因为计算机的发明而改变。Shneiderman指出，新的计算技术将更为关注人能做什么，但是人的天性和需要并不会因为计算机的发明而改变。 • 利用人与日常世界打交道时所形成的自然交互技能来获得计算机提供的服务更符合人的认知特点。

9. Why NUI? • GUI(MacOS、Windows操作系统） • GUI以桌面为隐喻，采用WIMP范式，但是频繁的菜单选择、按钮操作和键盘命令输入不符合人们的自然交互习惯，操作离散、界面布局复杂，认知负担重，对于非熟练的大众用户更是效率低下，WIMP界面已经成为制约用户生产力提高的瓶颈。

10. UI大师Alan Coope曾在著作《The Essentials of Interaction Design》中，在用户心理层次深刻地揭示了流状态：“深深的完全沉思状态”，经常产生“轻微的欢娱”，能够忘记时间的流逝，这就是自然用户界面的真实写照。

11. 自然人机交互的主旨。 • 软件交互应该促进和加强流状态，而不是打断或者干扰流状态。这就要求在用户的交互过程中，大部分交互行为都应在一种顺畅的状态下提供给用户。 • 普适计算之父Mark Weiser, 不可见的交互技术。所谓的不可见，是说工具不会干扰人们的认知，人们的注意焦点是任务，而不是工具。

12. 通过自然人机交互，用户可以利用语音、动作等自然表达，产生与计算机设备的无缝交互，无疑可以明显降低交互困难，提高交互效率。通过自然人机交互，用户可以利用语音、动作等自然表达，产生与计算机设备的无缝交互，无疑可以明显降低交互困难，提高交互效率。 • NUI研究的关键要点: 在于如何充分利用人自身的自然输入属性，尽可能避免对使用者的大量训练。 • N. Cross, "Natural intelligence in design," Design Studies, vol. 20, pp. 25-39, 1999. • C. Nass and B. Reeves, "Social and natural interfaces: theory and design," in CHI '97 extended abstracts on Human factors in computing systems: looking to the future, Atlanta, Georgia, 1997, pp. 192-193.

13. Input Device - Vision/Goals (1945-2015) ImmediateIntermediateLong-term • Combined speech recognition, character recognition • Pen editing • Heuristic programming • Ubiquitous computing • Time sharing • Electronic I/O • Interactive, real- time system • Large scale information storage and retrieval • Natural language understanding • Speech recognition of arbitrary users • Natural user interface • Internet of things (C2C, H2C, H2H, H2C2H)

14. Input Devices (overview) Sensor Devices 1. Spatial Position/Orientation Sensors • 2DOF (Mouse) • 3DOF (Microscribe, FreeD Joystick) • 6DOF (Polhemus Fastrack) 2. Directional Force Sensors • 5 DOF (Spacemouse) • 2 DOF (Joystick) 3. Gesture Recognition • Data Gloves 4. Eye Tracking 5. Speech Recognition Systems

15. The First Mouse (1964) Knee control Douglas Engelbart Years before personal computers and desktop information processing became commonplace or even practicable, Douglas Engelbart had invented a number of interactive, user-friendly information access systems that we take for granted today: the computer mouse, windows, shared-screen teleconferencing, hypermedia, groupware, and more.

16. Input Devices (1) Directional Force Sensors SpaceMaster

17. Input Devices (2) Gesture Recognition Dextrous Hand Master, Exos SUPERGLOVE, Nissho Cyberglove , 5th Dimension

18. Input Devices (3) Spatial Position/Orientation Sensors Polhemus InsideTrack (Magnetic Tracking) MicroScribe (Mechanical Tracking) FreeD Joystick (UltraSonic Tracking)

19. Input Devices (4) Visual Haptic Workbench The Visual Haptic Workbench consists of five hardware components. The dominant hand of the user experiences haptic feedback from the PHANToM, and the subdominant hand navigates through a menu interface via Pinch glove contact gestures. Head tracking is done with a Polhemus Fastrak receiver mounted on a pair of Stereographics CrystalEyes LCD shutter glasses. The subdominant hand can also be tracked with a separate receiver to facilitate more complex interaction paradigms. The audio subsystem gives the user additional reinforcement cues to clarify [see also http://haptic.mech.nwu.edu/intro/gallery/]

20. Historical Overview (1945-1995) [source: Brad A. Myers (1998). A brief history of human-computer interaction technology. Interactions, vol 5(2), pp. 44-54]

21. 提纲 • NUI的发展与基本理解 • 触控技术的历史 • 触控研究的重点与目的 • 触控技术的前景

22. Touch / Multi-touch • 触控技术是单点触控与多点触控技术的总称 • 自然人机交互中的重要组成部分 • 在操作时不用借助其它媒介可直接触摸 • Natural affordances • Offer a more compelling method to interact with a system than a mouse or other types of pointing devices

23. 根基 • 得益于Nakatani天才的软机器(Soft Machine, 1983)的想法 • 通过手对屏幕上显示的各类操作组件（如按钮等）的操作，实现一种可变的、动态的交互。 • L. H. Nakatani and J. A. Rohrlich, "Soft machines: A philosophy of user-computer interface design," in Proceedings of the SIGCHI conference on Human Factors in Computing Systems, Boston, Massachusetts, United States, 1983, pp. 19-23.

24. History of Multi-touch • 1982 - first multi-touch system called Flexible Machine interface developed in University of Toronto. • 1983 - Bell labs and Murray Hill published the first paper discussing touch-screen based interfaces.

25. Video Place • 1983 - Video Place /Video Desk (Myron Krueger) A vision based system that tracked the hands and enabled multiple fingers,hands, and people to interact using a rich set of gestures • Myron’s work had a staggeringly rich repertoire of gestures, muti-finger, multi-hand and multi-person interaction.

26. History of Multi-touch • 1984 Multi-touch Screen (Bell labs, Murray Hill NJ) - integrated with a CRT on an interactive graphics terminal, could manipulate graphical objects with fingers with excellent response time. • Myron’s work had a staggeringly rich repertoire of gestures, muti-finger, multi-hand and multi-person interaction. • Microsoft began research in this area..

27. Multitouch Tablet 1985 • Input Research Group, University of Toronto • Touch tablet capable of sensing an arbitrary number of simultaneous touch inputs, reporting both location and degree of touch for each

28. Sensor Frame (Carnegie Mellon University) • The device used optical sensors in the corners of the frame to detect fingers. • At the time that this was done, miniature cameras were essentially unavailable.  Hence, the device used DRAM IC's with glass (as opposed to opaque) covers for imaging. • It could sense up to three fingers at a time fairly reliably (but due to optical technique used, there was potential for misreadings due to shadows.

29. 1986 Bi Manual Input • 1986 Bi Manual Input (University of Toronto) - able to position/scale task and selection/navigate task

30. Apple Desktop Bus • 1986 Apple Desktop Bus (ADB) and the trackball scroller Init(Apple Computer/University of Toronto)‏ • E.g. The macintosh II and macintosh SE

31. Digital Desk 1991 • (Pierre Wellner,  Rank Xerox EuroPARC, Cambridge) - supported multi-finger and pinching motions (leads to moderm product e.g. Iphone)‏

32. Flip Keyboard - 1992 • Bill Buxton, Xerox PARC • A multi-touch pad integrated into the bottom of a keyboard.  You flip the keyboard to gain access to the multi-touch pad for rich gestural control of applications. • Graphics on multi-touch surface defining controls for various virtual devices.

33. History of Multi-touch • 1992:  Simon (IBM & Bell South) - A single-touch device relied on a touch-screen driven “soft machine” • 1992:  Wacom (Japan) - tablet that had multi-device/multi-point sensing capability

34. Starfire - inconceivable [5] • 1992:  Starfire (Bruce Tognazinni , SUN Microsystems) - Bruce Tognazinni produced an future envisionment film, Starfire, that included a number of multi-hand, multi-finger interactions, including pinching, etc.

35. Starfile Video

36. Bimanual 1994-2002 • Alias|Wavefront Toronto • Developed a number of innovative techniques for multi-point / multi-handed input for rich manipulation of graphics and other visually represented objects

37. Graspable/Tangible Interfaces 1995 • Input Research Group, University of Toronto • Demonstrated concept and later implementation of sensing the identity, location and even rotation of multiple physical devices on a digital desk-top display and using them to control graphical objects.

38. Active Desk 1995/97 • Input Research Group / Ontario Telepresence Project, University of Toronto • Simultaneous bimanual and multi-finger interaction on large interactive display surface

39. T3 – Wavefront 1997 • A bimanual tablet-based system • Utilized a number of techniques that work equally well on multi-touch devices

40. Haptic Lens 1997 • By Mike Sinclair, Georgia Tech / Microsoft Research • A multi-touch sensor that had the feel of clay, in that it deformed the harder you pushed, and resumed it basic form when released. A novel and very interesting approach to this class of device.

41. Fingerworks 1998 • Inventor: Newark, Delaware • Produced a line of multi-touch products including the iGesture Pad. They supported a fairly rich library of multi-point / multi-finger gestures.

42. Portfolio Wall 1999 • Alias|Wavefront,Toronto On, Canada • A digital cork-board on which images could be presented as a group or individually. • Its interface was entirely based on finger touch gestures that went well beyond typical touch screen interfaces.

43. Diamond Touch 2001 [7] • Mitsubishi Research Labs,Cambridge MA) • Capable of distinguishing which person’s fingers/hands are which, as well as location and pressure

44. SmartSkin – Sony 2002 • An architecture for making interactive surfaces that are sensitive to human hand and finger gestures. • This sensor recognizes multiple hand positions and their shapes by using capacitive sensing and a mesh-shaped antenna. • In contrast to camera-based gesture recognition systems, all sensing elements can be integrated within the surface, and this method does not suffer from lighting and occlusion problems.

45. Jeff Han • FTIR multi-touch. • Very elegent implementation of a number of techniques and applications on a table format rear projection surface.

46. Video of Han

47. 2007-2010 Iphone4

48. iPhone& MacBook Air

49. Microsoft Surface

50. Video