Complex XenDesktop use cases; common mistakes; tools and techniques for resolution

1. Complex XenDesktop use cases; common mistakes; tools and techniques for resolution Baptiste Duflos Manager, Escalation Services May 8th, 2012

2. Introduction and objectives

3. Tweet about this session with hashtag #SUM301and #CitrixSummit

4. Focusing on the major components of XenDesktop SQL Database Licensing WI Controllers VM Host (XenServer, Hyper-V, VMware) User AD VDAs

5. Deploying Controller Servers • All Controllers load balance session launch and VDA registrations • Configuring Controllers in an N+1 configuration allows for resiliency in case of a failure • All Controllers talk to the SQL database and should deployed as close as possible Controllers

6. Controller Server Scalability XD4.x: Hypervisor Pool Broker WI Hypervisor Pool Broker (ZDC) Broker Hypervisor Pool XD5.x: Hypervisor Pool Controller Hypervisor Pool WI Controller Controller(failed) Hypervisor Pool

7. Controllers – Scalability and Best Practices • Can overwhelm the hosting infrastructure with power state requests during peak times when many users logon and off. • You can throttle the amount of power commands sent per Controller with “MaximumTransitionRate” – default is 20, do NOT increase it won’t speed up power up times

8. Deploying SQL for XD Databases • XD 5 uses a single database with multiple schemas that map to XD services • Stored procedures are leveraged to reduce load on database • Database is critical to XD 5 – all Controllers have heartbeat to database SQL Databases

9. SQL – Database Mirroring • Database failure = Controller Failure • Only impacts new connections – existing or disconnected sessions not affected • Citrix recommends leveraging SQL Mirroring for fault tolerance • Mirroring sends transaction log from Principal database to the redundant database • If the principal database fails, user intervention is required to fail over the database • Citrix recommends using synchronous database mirroring with witness Mirror database Principal database Transaction log Witness Server

10. SQL – Best Practices • SQL transaction log is critical to monitor • Connection launches and idle desktops consume transaction log space • Use a fixed-size transaction log – auto-growth feature could impact response times • Leverage SQL Alerts when log reaches thresholds (recommend 50%) • Database failover tuning – adjust Controller heartbeat interval • Default heartbeat is 30secs and requires a SQL operation • Controllers unregister workers that do not heartbeat for over one minute • Controlled by Regkey: HKLM\Software\Citrix\DesktopServer\HeartbeatPeriodMs

11. Deploying Virtual Desktop Agents • VDA now uses “registry based” registration by default • Verify ports are open and firewall configured • Forward and Reverse DNS is required Virtual Desktop Agents

12. VDA – Scalability and best practice • Increase the Service timeouts if you expect periods with large amount of VMs rebooting – increase to 3 mins recommended • Optimize the logon process – improves desktop performance • Plan staged deployments and consider leveraging tools such like LoginVSI to perform scale and load testing before adding large groups of users to environment

13. Key points to remember • Controllers are resilient and scale well – keep deployments simple • SQL server plays pivotal role in infrastructure – protect it! • Make your end users happy – tune your VDAs for performance

14. Troubleshooting a session launch failure • Users were reporting they got an error when trying to launch their desktops • Admin noticed that intermittently VDAs would de-register at session launch Case Study Walkthrough

15. Environment overview • XenDesktop deployment with: • Web Interface 5.4 • XD 5.6 • SQL 2008 • VMWare 5.0 • Windows 2008 R2 Active Directory • Virtual Desktop Agent OS – Win7 32-bit • Citrix Receiver 3.1 XD5.6 Receiver • WebInterface5.4 VMWare 5.0 Active Directory SQL2008 VDAs

16. What did failure look like?

17. User attempts to start the session 1011011010 SSL 1011011010 SSL 1011011010 SSL 1011011101101110 11

18. Initial Troubleshooting • How often does it happen? • Any particular timeframe it happens? • Any specific users or images it happens more frequently with? • What changed? • Any event viewer messages?

19. What changed? Customer added second Controller for redundancy Customer had single server deployment Controller #1 VDAs WI VDAs register on Controller #1 Controller #2

20. Where do we start looking? • We found 4 interesting messages in Event Viewer: Warning – Event ID 2103: An unexpected exception occurred while the Citrix Broker Service processed an XML transaction. An incompatible client might be trying to access the XML service. Verify the compatibility of clients accessing the service. If this problem persists, reinstall the Citrix XenDesktop Controller. Error details: Transaction: 'RequestAddress' Exception Type: 'System.ServiceModel.Security.SecurityAccessDeniedException' Application Warning – Event ID 1060: The Citrix Broker Service failed to apply settings on the virtual machine 'KB-WIN7-01.get.services.citrite.net'.   Check that the virtual machine can be contacted from the Controller and that any firewall on the virtual machine allows connections from the Controller. See Citrix Knowledge Base article CTX126992.   Error details: Exception 'Access is denied.' of type 'System.ServiceModel.Security.SecurityAccessDeniedException'. Warning – Event ID 1039: The Citrix Broker Service failed to contact virtual machine 'KB-WIN7-01.get.services.citrite.net' (IP address ).   Check that the virtual machine can be contacted from the Controller and that any firewall on the virtual machine allows connections from the Controller. See Citrix Knowledge Base article CTX126992.   Error details: Exception 'Access is denied.' of type 'System.ServiceModel.Security.SecurityAccessDeniedException'. Warning – Event ID 1101: The Citrix Broker Service failed to broker a connection for user 'GET\atladmin' to resource 'KB-Win7-PW'. The Citrix Broker Service cannot find any available virtual machines. Please add more virtual machines to the site. If the problem is due to existing virtual machines not becoming available, see Citrix Knowledge Base article CTX126992.

21. Troubleshooting Methodology – verify environment • Check Firewall configuration • Active Directory mis-configuration • Forward DNS and Reverse DNS • Environmental checks: • Check for time skew • Default ports • Port conflicts

22. Troubleshooting Methodology – gathering data • Run Citrix Scout • TaaS beta • Enable logging on both Controllers • Run a CDFTrace

23. Citrix Scout / XD Collector (CTX130147) • Push button easy data collection system • Makes data collection and upload push button easy • Integrates data collected by Scout with the Citrix Tools as a Service (TaaS) backend • Simplifies data collection & analysis

24. http://Taas.Citrix.com/Beta Tools as a Service 1 2 3 Data Collection Quickly collect and upload your data Auto analysis health check Recommendations tailored to YOU

25. Enabling logging • Enabling Controller Service Logging - CTX127492 • CDF Control - CTX111961 Controller

26. Digging deeper – Controller log analysis CdsBroker:1:1:UpdateWorkerSettings configurationService.Set failed: System.ServiceModel.Security.SecurityAccessDeniedException: Access is denied. Server stack trace: at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message) Exception rethrown at [0]: at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg) at System.Runtime.Remoting.Proxies.RealProx" CdsBroker:1:1:UpdateWorkerSettings reject the worker (S-1-5-21-1123877020-465626563-3648135752-3586)" BrokerDAL:8:5:DAL >>> DeregisterWorker(S-1-5-21-1123877020-465626563-3648135752-3586, CommunicationFailure)" BrokerDAL:8:5:DAL >>> DeleteBrokeredSessionOnPrepareFailure(LaunchToken=54711b77-4fce-4edc-b31e-937bc7dca341, SinBin=True)

27. Using CDF Control • Parsing the CDF trace and enabling the expert shader feature allows us to quickly find exceptions which are typically highlighted in orange • High level failure is: “CdsWorkerAgent:8:5:UserAllowed: found no matching Controllers, access not allowed for user” • With CDFControl you can download the public TMF files which will allow you to parse the CDF trace and troubleshoot your issue

28. Digging deeper – CDF trace log analysis Initial trust failure: CdsWorkerAgent:8:5:CheckAccessCore: Calling delegate to provide SID list CdsWorkerAgent:8:5:CheckAccessCore: entered, have 1 trusted DDCs CdsWorkerAgent:8:5:UserAllowed: found no matching Controllers, access not allowed for user GET\KB-XD5-SP1-2$ S-1-5-21-1123877020-465626563-3648135752-3604 After worker Sin-Bin timeout: CdsWorkerAgent:1:1:Heartbeat to http://KB-XD5-03.get.services.citrite.net:80/Citrix/CdsController/IRegistrar rejected CdsWorkerAgent:2:1:EventLogManager decided to log event CDS_EVENT_WORKER_AGENT_HEARTBEAT_REJECTED of type Warning Re-Registered (after timeout expires): CdsWorkerAgent:2:1:Succesfully registered with http://KB-XD5-03.get.services.citrite.net:80/Citrix/CdsController/IRegistrar; starting heartbeats

29. Under the hood - VDA Session Launch explained Controller #1 VDA Worker flagged in DB as Ready VDA registers to Controller #1 Broker Service Desktop Service VDA Controller #2 ListOfDDCs=Controller #1 XML sends PrepareSession ticket to VDA Controller #2 is not in ListOfDDCs, VDA invalidates session launch request XML Broker XML broker queries DB for a ready worker XML Broker unregisters worker VDA checks ListOfDDCs to authorize PrepareSession WI Sends launch request to XML Broker XML Returns Error to WI WI WI Error returned to user Worker is placed in SinBin User launches session User SQL

30. Root Cause analysis • The customer added a second Controller to handle XML requests for redundancy • As soon as the new Controller was added to the WI XML failover list it was available to broker session launches by design • Since the new Controller was not added as an authorized trusted agent XenDesktop rejects the session logons • Workstation agent de-registers temporarily and then attempts to re-register

31. Resolution • DDCs that handle authentication must be authorized agents and added to “ListOfDDCs” registry value • CTX132536 outlines the registry key and how to define broker groups • Adding DDCs to WI XML failover list enables the ability for DDCs to handle session logons

32. Resources discussed

33. Optimal deployment recommendations • CTX124087 - XenDesktop Modular Reference Architecture • CTX127939 - XenDesktop 5 Database Sizing and Mirroring Best Practices • CTX123244 - High Availability for Desktop Virtualization - Reference Architecture • CTX120760 - XenDesktop - Design Handbook • CTX128700 - XenDesktop Planning Guide - XenDesktop Scalability • Whitepaper - Benchmarking Citrix XenDesktop using Login Consultants VSI

34. For More Information • CTX132536 - Worker Unregisters at Session Launch • CTX130147 - Citrix Scout • CTX111961 - CDFControl • CTX127492 - How to enable Controller Service Logging in XenDesktop 5 • CTX128075 - XDDBDiag: XenDesktop 5 Database Diagnostics • CTX128909 - XenDesktop 5 Logon Process and Communication Flow

35. http://Taas.Citrix.com/Beta Find out how to rev up environment maintenance See your Citrix pit crew in the expo hall with the Tools as a Service checkered racing shoes

36. We value your feedback! Take a survey of this session now in the mobile app Click 'Sessions' button Click on today's tab Find this session Click 'Surveys'

37. Before you leave… • Conference surveys are available online at www.citrixsummit.com starting Thursday, May 10 • Provide your feedback and pick up a complimentary gift at the registration desk • Download presentations starting Monday, May 21, from your My Organizer tool located in your My Account