Spiral 1 Design

1. Spiral 1 Design Joshua Horwitz Mackenzie Sweeney

2. Architecture Components • Server Communication: AMQP • Server-side Application: Python Tornado or NodeJS • RabbitMQ as message broker • Client-Server Communication: SockJS & RESTful API • Client-side Application: HTML, CSS, JS with jQuery • Browser Communication: PubSubJS

3. Spiral 1 Proposal • Design/Prototype client pub/sub environment with PubSubJS • Design/Prototype Map abstraction using CMAPI specification • Either modify Mapstraction or implement our own • Maps to support for spiral 1: • LeafletJS (at minimum) • Google Maps V3 • OpenLayers • Design/Prototype UI (for maps) • Design/Prototype Overlay Widget • Extremely useful to test interoperability with other widgets • Can incorporate this as an add-on to our widget • Keep access control in mind, but don’t prioritize

4. Client Pub/Sub • Implement using global pub/sub library that meets criteria: • JS-native • Widely used • Actively maintained/developed • Our recommendation is PubSubJS // Basic pub/sub operations PubSub.subscribe(‘map.overlay.create’, callback); PubSub.publish(‘map.overlay.create’, payload); // Can also use jQuery syntax $.pubsub(‘subscribe’, ‘map.overlay.create’, callback); $.pubsub(‘publish’, ‘map.overlay.create’, payload)

5. Map Abstraction Layer • Goal: 0% code-rewriting, 100% interoperability • Possible starting point is Mapstraction (BSD License) • Fork and extend • Or create our own: take what’s useful and improve • Implement CMAPI over all supported maps • At a minimum we should support • LeafletJS (uses OpenStreetMaps) • Google Maps V3 • OpenLayers • Should also support multiple data formats • KMLGeoJSON

6. How Does it Work? • Create a map template interface • For all maps, implement the same interface (think Java) • User selects an underlying map to use (LeafletJS, Google, etc) • Based on user choice, load in appropriate module • Object guaranteed to have defined template • Can subscribe to appropriate channels with known callbacks • Essentially, its polymorphism • Benefits • No shim required: no unnecessary parsing • One handler, one time • Does not rule out subsequent reloads (swap underlying map)

7. User Interface • Much of the UI will be provided by the maps provided • Custom widgets (like overlay) should leverage: • HTML5 • CSS3 (responsiveness for different clients) • JS/jQuery • Priorities • Modern feel! • High degree of usability (learn from JC2CUI widget mistakes) • Persistent storage in browser • Get fancy later • Could create our own Grid widget (industry folks hate JC2CUI’s) • Provide cool analytics on mapped data (Jake/David )

8. Server Pub/Sub (looking forward) • Leverage RabbitMQ • Single exchange routes published messages to rooms • All messages for one room end up at one queue • Provides a pure FIFO structure to receive messages in order • Producers (users): • publish directly to the exchange via SockJS and the RabbitMQ Web-STOMP plugin • Consumer (web server): • handles the user subscriptions and sends the data to users via SockJS

9. Consumer Model • 2-layer consumer model • Web server • Browser (user) • Walkthrough • Web server consumes messages from room queues • Gets list of users in room from DB • Broadcasts message to all users over SockJS connections • Browsers (users) then receive messages • Channel parsed from message; appropriate callback executed

11. Benefits of Consumer Model • Obvious benefits • Establishes central authority on room state • Update new users who join room • Recover from disconnects • New features • Collect data on publishing patterns • Save existing session state for quick restore later • This approach has been proven in the commercial world • It works! • No need to modify current CMAPI channel payloads! • HuffingtonPost Live pumps out over 20 million messages/week

12. AMQP • Developed to solve the messaging interoperability problem in the financial industry • Used now by 100s of critical systems • Open standard • Highly interoperable • Use-cases • Transient pub/sub distribution • Reliable request-reply transactions

13. RabbitMQ • Market leader MOM for AMQP • Most popular choice on EC2 • Bindings for multiple languages • Java, Ruby, Python, C, C#, Erlang, Node, Perl • Multi-protocol Support • AMP 0-9-1, 0-9 and 0-8 • AMQP 1.0 • MQTT, XMPP, SMTP, HTTP, STOMP

14. RabbitMQ • Built-in RESTful API & Web-STOMP plugin • The creators have web messaging in mind! • Reliability – can persist data and recover from failures • Built-in clustering and transparent scaling • Self-healing capabilities and extensive tooling/configuration • Built-in access control techniques

15. RabbitMQ Web-stomp • A simple bridge exposing STOMP protocol over emulated HTML5 websockets • Makes it possible to use RabbitMQ from web browsers • Comes with RabbitMQ • Used in industry (see previous huffington post example) • They handle over 20 million messages/week with web-stomp • EXTREMELY SIMPLE to use

16. Web-STOMP Code // Stomp.js boilerplate varws = new SockJS('http://' + window.location.hostname + ':15674/stomp'); var client = Stomp.over(ws); send = function(data) { client.send('/topic/bunny', {}, data); }; varon_connect = function(x) { id = client.subscribe('/topic/bunny', function(d) { // do some stuff }); }; varon_error = function() { console.log('error'); }; client.connect('guest', 'guest', on_connect, on_error, /');

17. So what is this SockJS? • A wspolyfillthat provides a ws-like JS object in ANY browser • Automatic fallback if the channel can’t support websockets • SocketIO does not do this • Also works in browsers behind ws-unfriendly proxies • If it doesn’t work, your network broke the Internet • Well documented scaling and load balancing techniques • Can utilize sticky sessions (JSESSIONID) or prefix-based LB • Used in critical industry systems

18. Server Application • Must support SockJS • Our recommendation is Python/Tornado • 2nd option recommendation is NodeJS/Express • The approach we are outlining has been proven in the commercial industry as a viable solution • http://www.erlang-factory.com/upload/presentations/791/HuffingtonPost.pdf • Extremely easy to implement • simple prototype has been developed to proof feasibility