Touch Seen Visual Board Game Platform

1. Touch SeenVisual Board Game Platform Computer Vision Workshop, Fall 2010 EldadZinger, Oren Kishon, Ehud Fisher, Allon Freedman

2. Agenda • Overview • Architecture • High Level • Server • Client • Camera Module • Network Protocol • Game Plugin Platform • Client-Camera Interaction • Game move detection algorithm • Demonstration Touch Seen

3. Overview Touch Seen

4. Touch Seen Application framework for playing board games over a computer network using a real physical board Moves are detected by a camera and transmitted over the network Provides a simple API for development of 3rd party games which can be uploaded to the platform in the form of plugins Touch Seen

5. Features Input using camera or any other standard input (keyboard, mouse etc.) Multiple clients connected simultaneously Multiple games played simultaneously All currently connected users displayed in client Option to challenge another user to a game Textual chat between players Touch Seen

6. Features Open API for game development Pluggable games Simple text based protocol Network based on ActiveMQ which supports multi language clients Touch Seen

7. Architecture Touch Seen

8. High Level Server Client Client Touch Seen

9. Server Two logical functions: • Messaging – distributing Move and Chat messages between users • User & Game Management: • User DB • User presence • Game state Touch Seen

10. Server • Messaging server based on Apache ActiveMQ • ActiveMQ – powerful open source Messaging Server supporting JMS 1.1, J2EE 1.4, multiple cross language clients and protocols and more http://activemq.apache.org/ • An ActiveMQ server is run as an embedded broker within the Touch Seen Server Touch Seen

11. Server JMS • A specification that describes a common way for Java programs to create, send, receive and read distributed enterprise messages • loosely coupled communication • Synchronous & Asynchronous messaging • Reliable delivery • A message is guaranteed to be delivered once and only once • Outside the specification • Security services • Management services Touch Seen

12. Server JMS Messaging Domains • Point-to-Point (PTP) Message Queues - each message has only one consumer • Publish-Subscribe systems Message Topics - each message has multiple consumers Touch Seen

13. Server JMS Message Consumption • Synchronous • A subscriber or a receiver explicitly fetches the message from the destination by calling the receive method • The receive method can block until a message arrives or can time out if a message does not arrive within a specified time limit • Asynchronous • A client can register a message listener with a consumer • Whenever a message arrives at the destination, the JMS provider delivers the message by calling the listener's onMessage() method • All messages in Touch Seen use the asynchronous method Touch Seen

14. Server Msg Msg consumes sends Queue acknowledges Client1 Client2 JMS Messaging Domains – Queues Touch Seen

15. Server subscribes Msg delivers publishes subscribes Topic Client1 Client2 Client3 delivers JMS Messaging Domains – Topics Touch Seen

16. Server Connection Factory creates Connection Message Producer Message Consumer creates creates Session receives from sends to creates Msg Destination Destination JMS Model Touch Seen

17. Server – Class Diagram Games Users Touch Seen

18. Server Threads ServerController & NetworkController each operate in separate threads This ensures that server operations never block the network thread so it can always be available for sending/receiving messages Management The server has no UI Management is done using JMX (e.g. with jconsole) utilizing both APIs provided by ActiveMQ and custom API’s designed specifically for this server Touch Seen

19. Client Strict modular design based on MVC Comprised of several independent modules Each module has a controller class ClientController acts as central controller for entire application All communication to and from a module is done through its controller The central controller connects to each of the module controller (module controllers don’t interact directly) Touch Seen

20. Client • The advantages of this design: • Modules are entirely decoupled from each other • Each module can be developed and tested independently • Communication between modules is done in one place – easy to maintain and debug • Modules can easily be replaced without affecting the rest of the application – only the central controller “knows” all the modules Touch Seen

21. Client – Class Diagram Touch Seen

22. Threads The ClientController, NetworkController and CameraController all operate in separate threads This ensures that the Network and Camera threads are never blocked by other modules within the client All Objects in the View operate within Swing’s EventDispatcher thread The ClientController and Model also operate within the EventDispatcher thread to avoid synchronization problems with the View Touch Seen

23. Network Protocols Touch Seen

24. Login username password Login Queue success notify online Players Topic request players Players Request Queue players Touch Seen

25. Game New Game challenge challenge Challenge Queue accept accept notify notify Players Topic Touch Seen

26. Game Game Playing move move Game Topic move message message Game Chat Topic message Touch Seen

27. Game Game End quit quit Game Topic quit notify online Players Topic Touch Seen

28. Logout username Logout Queue notify offline Touch Seen

29. Game Plugin Platform Touch Seen

30. Game SDK • Games are developed using a provided SDK • Developers must implement only two abstract classes: • GameInfo • Flyweight which acts as a factory for creating instances of the game • Provides common properties: name, icon etc. • Game • Represents the game itself Touch Seen

31. Game SDK The server and client controllers interact with the game using the Game class It’s recommended (although optional) that this class include the game’s logic Game must provide a JPanel which acts as the game’s UI Touch Seen

32. Game Installation A game should be packaged as a .jar file and placed within the lib/games folder of the client and server A configuration file must be edited with the fully qualified name of the GameInfo class At startup, the client and server use a custom ClassLoader to load all the games dynamically A future release could include automatic deployment of new games to clients and installation/uninstallation without restart A simple (yet incomplete) example of Chess is included Touch Seen

33. Client-Camera Interaction Touch Seen

34. Client-Camera Interaction The Touch Seen desktop client is written in Java The camera module is based on OpenCV and written in C Interaction between the two is done via a local TCP socket The protocol is a simple text based protocol which simplifies debugging When the user starts a game, the client starts the camera process and waits for a socket connection for a specified timeout (externally configurable) If the initialization sequence completes successfully the camera can be used to play Alternative input devices (e.g. a mouse) can be used if errors occur during the initialization sequence or if moves aren’t detected successfully Touch Seen

35. Game Move Detection Algorithm Touch Seen

36. Detection algorithm Image subtraction: Touch Seen

37. Detection algorithm (1) • Initialization: • Detect chessboard • Find prospective transformation (3 x 3): rectify • Rotate coords to user’s point of view • For each new image captured, do: • Apply prospective transformation (rectify) • Subtract image from previous image – to detect who moved / captured by opponent player • Subtract image from empty board image – to detect who actually cleared his location Touch Seen

38. Detection algorithm (2) Dealing with noise • Noise: intensity only gets higher values • Filter1: image is a MIN of 20 frame captures • Filter2: Time-adaptive noise threshold: • Capture two images each time, 1 sec apart • Subtract to get noise-only image • Calculate average value for square • Find max among squares (empirically ~5) • Thresh = max + const (=2, heuristics) Touch Seen

39. Detection algorithm (3) Square change detection • Less sensitive change detection: • Among squares who passed threshold, take max. • <Used for move detection> • More sensitive change detection: • For each square, sum value of pixels, regardless of threshold. Take Max. • <Used for finding captured piece> Touch Seen

40. Detection algorithm (4) Evolution of development At first – simple subtraction algorithm (gray): hard-coded thresh, one background image Use RGB space – track each piece by its color. Each piece physically had a unique color Use HSV space – not good when dealing with B&W backgrounds Back to gray – smarter algorithm: min filter, multiple backgrounds, adaptive threshold Touch Seen

41. Demonstration In addition to the live demonstration we have provided the following documentation: System Requirements Specification describing planned features and architecture This presentation which provides the final design implemented User Manual Complete Javadoc for the Client, Server and Game Development SDK including UML diagrams (embedded within the Javadoc) Fully documented code Source code for Client, Server, SDK, Camera module and example game (Chess) Installation instructions Touch Seen

42. Thank you… Touch Seen