SharePoint Patching: Notes From the Field

1. SharePoint Patching: Notes From the Field Neil Hodgkinson Senior Premier Field Engineer Microsoft UK Ltd

2. About Neil.. Microsoft UK Ltd sharepoint.microsoft.com/blogs/fromthefield neil.hodgkinson@microsoft.com Senior Premier Field Engineer EMEA Technical Lead : Office Servers EMEA Lead for MOSSRAP Before that CSC GIS Web Services Specialist

3. Ask The Experts Panel Text numbers 1. IT PRO � 07891 100640 � 2. DEV � 07790 108093 � 3. IW � 07790 132855 � 4. BA � 07989 359425

4. Agenda

5. So what is patching all about?

6. Terminology

7. Terminology

8. Hotfix COD �Critical on Demand Public Hotfix e.g. US DST Hotfix 941422 Cumulative Update Since December *� rolling 2 month basis Broadly affecting support issues Service Pack Service Packs include all of the updates for MOSS plus deliver important customer-requested stability and performance improvements, while incorporating further enhancements to user security. Type of Updates Released

9. Build Numbers

10. Check Your Build � How?

11. Check your build How to confirm the deployed version(s) of WSS/MOSS in your farm

12. Check Your Build � How?

13. Patching Post IU

14. How it used to be...

15. The December CU and Beyond ... Simple again to show the improvement in this with all the updates now being available in 2 packages So admins wanting to stay completely up to date have a clear path to get there. Remember to say that the individual updates are still available if admins want to patch just the �broken parts� Simple again to show the improvement in this with all the updates now being available in 2 packages So admins wanting to stay completely up to date have a clear path to get there. Remember to say that the individual updates are still available if admins want to patch just the �broken parts�

16. So what do I get for my effort?

17. What�s in the MSP?

18. Dismantle an update How to understand the payload of your Update

20. But what do I really need?

21. Updating To

22. Which build should I be on? Minimum required Microsoft recommended Latest and greatest "If it isn�t broke don't fix it attitude" is becoming less prevalent (based on MS Case Histories) Culture Shift

23. Version Distribution

24. Upgrade Scenarios

25. Typical Scenarios

26. What is the best way to build it? Slipstream It Any package can be slipstreamed - extract the MSPs into the update folder Benefits of fixes in the provisioning and config wizard code paths. When you configure CA it is configured with the fixes and updates already in place. Long term is no different to manual updates but is more efficient SP1 is Requirement for Install on Windows Server 2008

27. Adding a new server to an existing farm: Replacement Scale-out process New server must be same version as existing servers How do we achieve this? Slipstream installation Manually install patches Do the above before running configuration wizard to join the farm

28. Patching an existing server in an existing farm: New Microsoft recommendations To correct an error that has been experienced (per a KB article) To be running the most recent code How do we achieve this? Manually install patches Do the above before running configuration wizard to join the farm

29. Slipstream Creation Requirement for deploying on Windows Server 2008 Deployment guides Windows SharePoint Services 3.0 http://technet2.microsoft.com/windowsserver/WSS/en/library/91649a7e-6b5a-4e5a-9ee5-51951f4b857f1033.mspx Office SharePoint Server 2007 http://technet2.microsoft.com/Office/en-us/library/f484f5f2-35bb-4d70-bf56-dd1c4c287c721033.mspx

30. Slipstream Creation

31. Planning and Upgrading

32. Pre Upgrade Steps Take a complete farm backup and verify it restores It is not supported to restore an earlier version onto a newer version e.g. RTM to SP 1 Backup any SharePoint out of box files you may have customized Run orphan repair tool, http://support.microsoft.com/kb/923904 Defragment databases, http://support.microsoft.com/kb/943345 Ensure adequate disk space exists on each server Installation of the new binaries, new search overhead, and upgrade logging will take disk space Run a simulation on staging environment

33. Load balance sites across databases if needed Whitepaper released at SP 1, Performance Recommendations for Storage Planning and Monitoring, http://go.microsoft.com/fwlink/?LinkID=105623&clcid=0x409 Verify no upgrade Timer Jobs are currently running In place upgrade needs to complete before installing an update Gradual upgrade needs to complete current site upgrade before installing an update Take a full farm backup immediately after you know the update has applied successfully In case of a catastrophic problem shortly after update applied it will be easier to restore than restoring the RTM backup

34. Deploying Overview All updates follow the same steps and procedures Make sure the account you will be installing the update with has adequate permissions Mechanics Install the update on the SharePoint server(s) Installs the updated binaries Run SharePoint Configuration Wizard or psconfig.exe Updates the web applications, databases, and versions The database upgrade is the longest running piece Until the database upgrade is complete access to content is blocked to prevent corruption

35. Deployment Options Single server Basic installation Automatic updates In place installation for a farm Install the update on each server until the following dialog appears Click OK on the dialog on the first server Wait until it has completed Continue to each of the remaining servers

36. Deployment options � DB attach Document all databases assigned to each web application Remove all of the databases from the web applications Install the update on each server following the �in place� method Re-attach the content databases to respective web applications Note: Attaching the content database has been reported to be faster and give more flexibility and control than normal �in place� upgrading of a farm

37. Minimising Downtime � Parallel Upgrade High performance db attach MSIT Scenario: 4 Terabytes of data to upgrade 12,000 site collections to upgrade 48 hours to complete the upgrade Upgrade Procedure: Create the 4 farms (Identical target patch version) Backup databases, verify and detach Attach databases to upgrade farms (in parallel) Verify upgrade has completed successfully Detach databases from upgrade farms (in parallel) Attach upgraded databases to production farm Complete with quadrupled throughput!

38. Minimising Downtime Duplicate Read-Only Farm Internet Scenario: 24x7 Internet Presence Required Upgrade Procedure: Create 2 farms (Identical original patch version) Backup databases Attach databases to target farm Change routing / DNS Verify internet access to �Read Only� farm Upgrade production farm Switch back to Upgraded farm Complete with continued uptime!

39. Sequence of Installation WSSv3 Updates must be applied before the MOSS 2007 Updates FAQ � Can I install WSS and MOSS updates and run the configuration wizard once?  FAQ � Can I uninstall any update if there are problems? Language Pack updates For MOSS it is necessary to install the Language Pack SP1 update for both WSS and MOSS At MOSS RTM we only installed the MOSS Language pack because it also contained the WSS Language Pack files

40. Upgrade Order

41. Verify Update was Successful The Upgrade Log file Finished upgrading SPFarm Name=<Configuration Database> In-place upgrade session finishes. Root object = SPFarm=<Configuration Database>, recursive = True. 0 errors and 0 warnings encountered. If you cannot find the entries look for Exception or Error, correct the problem and run the configuration wizard or psconfig.exe again Check the Build Version

42. Monitoring and Troubleshooting

43. Monitoring and Troubleshooting Most support issues reported on deploying updates relates to failures running the Configuration wizard SharePoint Products and Technologies Configuration wizard psconfig.exe

44. Log files PSCDiagnostics log files, e.g. PSCDiagnostics_1_21_2008_14_27_10_84_453374783.log Upgrade log files, e.g. Upgrade.log Event Logs PSCDiag and Upgrade log files are stored in: c:\Program Files\Common Files\Microsoft Shared\web server extensions\12\LOGS Monitoring and Troubleshooting

45. Monitoring and Troubleshooting

46. Sequences can repeat in certain scenarios Multiple content databases Multiple web applications If the SPTimerv3 account can not write to the default Upgrade.log, it may write to  \Documents and Settings\SPTimerv3 account\ Local Settings\Temp\Upgrade.log The Upgrade.log file is cumulative Monitoring and Troubleshooting

47. When it all goes wrong

48. Issue: �Server Error� when accessing SharePoint site Cause: WFEs are upgraded but Databases are not Resolution: Detach/reattach version mismatched database(s). Common Failures

49. Issue: PSConfig fails with inadequate install permissions Cause: Not logged in with adequate permissions to install the update Overall #2 SP1 deployment failure Resolution: Ensure necessary permissions when deploying any update Administrators on the WFEs Administrators on the SQL Server or granted the database role db_owner on all SharePoint databases Common Failures

50. Issue: PSConfig fails after content database attach Cause: [NewWebParts] [12.1.6301.0] [DEBUG] [1/10/2009 8:15:40 AM]: Begin Upgrade()........[SiteSequence] [ERROR] [1/10/2009 8:15:40 AM]: Unknown server error number: d Resolution: Disable ForeFront Security for SharePoint Run PSconfig, or detach/reattach content databases Re enable ForeFront � Don�t forget this step Common Failures

51. Issue: PSConfig fails at Step 8 Cause: [SPManager] [ERROR] [12/19/2008 10:46:02 AM]: Upgrade [SPSearchDatabase Name=OSST_SEARCH Parent=SPDatabaseServiceInstance Name=sql08,1433] failed. ... [SPManager] [ERROR] [12/19/2008 10:46:02 AM]: Object reference not set to an instance of an object. Resolution : Corrupt dll in Dec CU [Build 6334] Common Failures

52. Issue: PSConfig Fails at Step 8 Cause: " SPWebTemplateSequence] [ERROR] [12/19/2008 12:08:53 PM]: Template SPSSITES#0: Exception occurred in activating features in site with URL "<http://servername/SiteDirectory>" ... ... Skipping this site for template upgrade. Exception: System.NullReferenceException: Object reference not set to an instance of an object. Resolution: The 'require content approval' feature must be activated on the Sites list if you are using a web application with a publishing template for the site collection Common Failures

53. Issue: A error during the installation of a hotfix package, which contains the S2 search administration (build 6314+) Cause: [SPWebTemplateSequence] [ERROR] [5/21/2008 9:57:56 AM]: Template OSRV#0: Exception thrown in activating SPSite scoped features for SPSite with URL "http://w2k3sp:5000/ssp/admin" (Id=10fd2ac1-fd56-475d-861f-c3b27af34a5b). ... Skipping this SPSite for template upgrade. Exception: System.InvalidOperationException: Feature '2b1e4cbf-b5ba-48a4-926a-37100ad77dee' is not installed in this farm, and can not be added to this scope. Resolution : The s2searchadmin feature had not been installed/activated Common Failures

54. Final Thoughts... Understand what you have Understand where you want to be Plan the upgrade Be prepared And one more thing... 100% Uptime is a Myth � Would you like to have a heart transplant and try to do your day job at the same time?

55. Thank you for attending! Please be sure to fill out your session evaluation!