Mesh Services Architecture And Concepts

1. BB06 Mesh Services Architecture And Concepts  Abolade Gbadegesin Principal Software Architect Microsoft Corporation

2. Live ServicesUser-centric data and services with more than 460M Users Users Devices Applications Synchronization Mesh Services Identity Directory Storage Communications and Presence Search & Geospatial

3. Agenda • User Experiences • Concepts • Architecture • Planning and Deployment • Implementation • Operations, Experiences, Lessons Learned • Future Work

4. demo Using the Live Mesh software

5. Key Capabilities • Provisioned state • User, devices, applications, relationships • User-data storage • Feeds and content • Feed synchronization and content replication • Communications and presence • Device presence and remote UI access • Peer connectivity • Notifications • Single channel for asynchronous push • Signaling channel for session establishment Accounts User-data storage Communications & Presence Notifications

6. Provisioning In The MeshConceptual elements builds on Live ID authentication to capture user-device relationships provisioned during web sign-up , identity 1 device claim 3 device 2 provisioned during software install

7. Understanding REST In The MeshResources, collections and links scope for grouping resources and tracking changes 2 collection Uri EntityTag SubscriptionUri … 3 represented as feeds and items resource Uri LastUpdateTime … resource 1 metadata common to all resources link Uri Label … primitive for representing & navigating relationships resource 4

8. Provisioned State ServicesConceptual view and building blocks HTTP Live ID Client Runtime Library Live ID Service Mesh Service Live Mesh Client Accounts Accounts HTTP (local cache) users, devices, apps

9. Folder Synchronization In The MeshConceptual elements items in data feed represent files 1 local file system mesh object mesh data feed data feed mapping , mesh object defines permission scope for sharing feeds 3 2 ‘maps’ data feed to a file system location

10. Understanding FeedSync In The MeshEnabling bidirectional feed synchronization <rss version="2.0"> <channel> <title>To Do List</title> <description>A list of items to do</description> <link> http://example.com/partial.xml </link> <item> <title>Buy groceries</title> <description>Get milk, ... </description> <sx:sync id="item_1_myapp_2005-05-21T11:43:33Z" updates="3"> <sx:history sequence="1" when="2005-05-21T09:43:33Z" by="a9"/> <sx:conflicts ...> <item> ... </item> </sx:conflicts> </sx:sync> </item> </channel> </rss> version vector and conflicts for each feed item

11. Synchronized User-Data Storage ServicesConceptual view and building blocks Mesh Service Live fx Client User-data storage 010010010100111100 User-data storage HTTP, FeedSync (local replica) feeds, content, permissions Mesh folders app handles mesh folders , local file system Scalable storage

12. Mesh Services Architecture Mesh Service Mesh UX Live ID Service Accounts Live fx Client Notifications Accounts Live ID Client Runtime Library Messenger Relay Service Communications & Presence Notifications User-data storage Communications & Presence Messenger NAT Traversal 010101 User-data storage Scalable storage Local file system Services fabric Microsoft Update

13. Mesh Services ArchitectureGetting from concepts to implementation • Derive workloads from dominant application flows • Identify key scalability factors • Decide on the partitioning of state • Assign machine roles across tiers

14. Mesh Services PlanningWorkload analysis: Startup and file updates Live fx Client Mesh Accounts Mesh User-Data Storage Startup update local cache subscribe for change notifications synchronize local replicas subscribe for change notifications File change synchronize feed upload updated files

15. Mesh Services PlanningWorkload analysis: Content replication Live fx Client Mesh Notifications Mesh User-Data Storage File change continued fire change notifications deliver change notifications Content replication synchronize local replica query availability of updated files Mesh Communications and Presence query availability of peers fetch updated files from peers relay or direct connection

16. Mesh Services PlanningScale-out strategy • Partition by user, device, and mesh object • Use soft state to minimize I/O load • Leverage HTTP 1.1 semantics for caching, change notification, and incremental state transfer • Leverage client-side resources for holding state • Leverage peer connectivity for content replication

17. Mesh Services PlanningPartitioning resources each resource lives in a ‘home’ datacenter each resource has a stable canonical URI based on its resource identifier mapping a resource identifier to its ‘home’ datacenter requires a lookup in a replicated database https://accounts.mesh.com/Identities/FKEXBSLJR3VUXEXMTPSSOVO6GM/DeviceClaims/CSOW2TEBYUSE5BD3I5QOS724UU https://accounts.clus9.bay.mesh.com/Identities/FKEXBSLJR3VUXEXMTPSSOVO6GM/DeviceClaims/CSOW2TEBYUSE5BD3I5QOS724UU each resource has a location URI that routes requests directly to its ‘home’ datacenter requests to the ‘wrong’ datacenter get redirected, and clients cache the redirect URI for future use

18. Mesh Services PlanningGeneral service design principles • Commodity hardware • Partitioning for scaling out, redundancy for availability • Loose coupling across roles • Xcopy deployment and configuration • Fail-fast, recovery-oriented error handling • Self-monitoring and self-healing

19. Mesh Services DeploymentInfrastructure and roles Front end role Back end role Building block Account APIs front end roles Services fabric User-Data Storage Content upload & download Web Desktop Scalable storage Services fabric SKU: 2P Dual Core 8GB RAM 4x 500GB SATA II HDD Windows Server Device presence store scale unit partitioning units Subscription store back end roles Notification queue store Dictionary state store commodity hardware xcopy deployment automated rollback self-monitoring fail-fast recovery self-healing Partitioning & Recovery Manager infrastructure roles Watchdogs

20. Mesh Services DeploymentScale unit layout Front end role Back end role Building block soft state stores Subscription store Accounts Scalable Storage Partitioning & Recovery Manager User-Data Structured Storage User-Data Blob Storage reliable state stores Watchdogs Dictionary state store Device presence store Notification queue store soft state stores Web Desktop User-Data Storage API Content upload & download Accounts API Communications & Presence Notifications API Services fabric building block services Live ID Messenger Relay Hotmail

21. Mesh Services ImplementationSoft state partitioning and recovery hash nodes & resources to the same output space node ‘nearest’ a resource is responsible for it Node 1 Node 2 {ECCA8793-3208-48A9-A1A1-038D0AD704C5} /MeshObjects/FJ3GZXUNDC4EPPPPUMQ2QNCG6I/Subscriptions/GLW6BC6HRWFU5LEW7HTX5FPC3Y consistent hashing node loss tracking Device presence store Partitioning & Recovery Manager Notification queue store resource owner nodes Owner library Owner library Communications and Presence Notifications API Lookup library Lookup library resource client nodes owner library obtains leases lookup library queries assignments

22. Mesh Services ImplementationCommon software stack Mesh service code Building block code multiple processes mesh service process singleton process Mesh services runtime Mesh service application Tracing & log collection Services fabric controller AP configuration Mesh services host process NETFX 3.5 SP1 Windows Server SMB.sys

23. Mesh Services ImplementationFront end software stack Mesh service code Building block code multiple processes mesh service process singleton process TLS/TCP handlers Mesh resource handlers Mesh content handlers Mesh services runtime Tracing & log collection Services fabric controller Accounts Notifications TCP notification streaming Communications & Presence Soft state lookup User-data content upload & download Scalable storage client User-data storage TLS/TCP stream dispatcher Subscriptions HTTP request dispatcher AP configuration Mesh services host process NETFX 3.5 SP1 WCF syndication Windows Server HTTP.sys sockets SMB.sys

24. Mesh Services ImplementationBack end software stack Mesh service code Building block code multiple process mesh service process singleton process Mesh resource managers Mesh services runtime Tracing & log collection Services fabric controller Device presence manager Notification queue manager Soft state owner/lookup Dictionary store manager Scalable storage client Subscription manager Subscriptions Partitioning and recovery manager Work queues Sockets dispatcher AP configuration Mesh services host process NETFX 3.5 SP1 Windows Server sockets SMB.sys

25. demo Monitoring The Mesh Services

26. Mesh Services OperationsExperiences and lessons learned On availability: Design for loosely coupled dependence on building blocks Diligently validate client/cloud upgrade scenarios Invest in pre-production stress and functional coverage in environments that look like production Design for throttling based on both dynamic thresholds and static bounds

27. Mesh Services OperationsExperiences and lessons learned On monitoring: Continuously refine performance counters, logs, and log processing tools Monitor end-user-visible operations (Keynote) Build end-to-end tracing across tiers Self-healing is hard: Invest in tuning watchdogs and thresholds

28. Mesh Services OperationsExperiences and lessons learned On deployment: Deployments every other week, client upgrades every month Major functionality roughly each quarter Took advantage of gradual ramp to learn lessons early

29. Mesh Services FuturesExtending the mesh cloud personal and enterprise user-data storage offerings: on-premises software and online service federate user authentication with other identity providers (e.g. Active Directory) You saw this working today: Microsoft Services Connector Stay tuned! synchronize user-data storage with non-Microsoft services (e.g. Smugmug) You saw this in the Sync Framework session

30. Microsoft Services ConnectorFederating identity and policy Azure Platform Services Enterprise Apps ISV Apps Live Services Online Services Windows Azure Live Mesh CRM Online Live Identity Provider Microsoft Federation Gateway Soon: enterprise credentials in Live Mesh Future: enterprise policy on Mesh Services Microsoft Services Connector Enterprise On-Premises Active Directory Live Mesh Desktop Your Apps Sharepoint Exchange

31. Sync Framework and Live Framework Synchronizing clouds Mesh Services Mesh Services SyncFX store providers Future: Mesh Services as a sync hub for your apps Cloud Sync Accounts Notifications Communications & Presence Live Framework Sync apps User-data storage 010101

32. Summary • The Mesh Services deliver a comprehensive and compelling suite of key capabilities • You can build on the Mesh Services today using the Live Framework and Sync Framework • Your feedback will help define the roadmap for delivering personal and enterprise offerings of the Mesh Services

33. Related Sessions and Resources • Live Mesh • http://www.mesh.com • http://blogs.msdn.com/livemesh • Live Framework and Mesh Services • BB04: Live Services: A Lap Around the Live Framework and Mesh Services • BB34: Live Services: Notifications, Awareness, and Communications • Sync Framework • http://msdn.microsoft.com/sync • TL30: Microsoft Sync Framework Advances

34. Evals & Recordings Please fill out your evaluation for this session at: This session will be available as a recording at: www.microsoftpdc.com

35. Q&A Please use the microphones provided

36. © 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.