SHAREPOINT 2007 PERFORMANCE TUNING MUST-HAVES

1. SHAREPOINT 2007 PERFORMANCE TUNING MUST-HAVES or..."What you should be doing to help your mature-as-it's-ever-going-to-be SharePoint 2007 farm through to SP2010“presenter:Mark VogtVOGTLAND CONSULTING2010.03.18

2. About The Presenter… • Mark Vogt, Mark Vogt, Ltd – SharePoint Consultant, PM & EA • Presenter Bio: • BS: Aerospace Engineering - U of MN IT • MS: Mechanical Engineering & Robotics (ABT) – U of MN IT • MS: Computer Science/AI - IIT • Consulting in KM & portal technologies since 1995 • Roles actively include developer, PM, architect & manager • Worked with SharePoint technologies since it  came out in 2000 • Runs blogs & videos focusing on SharePoint, Consulting & “The Big Picture”... • LOVES SharePoint • Sees SharePoint as one-of-a-kind enabling technology with no equal • SharePoint Purple Belt

3. Target Audience: SharePoint Green Belts… Most CSPUG Attendees are Green Belts - people experienced in using SharePoint for months/years, possibly experts in certain aspects of SharePoint, but NOT experts in all… • CSPUG Chairs  • Presenters   • Sponsors    • Visitor/Novices     • White Belts      • Green Belts                   • Purple Belts    • Brown Belts   • Black Belts 

4. Deliverables: from this Presentation… • Overview of Situation for you & your colleagues • Definitionsfor you & your colleagues • Explanations for you & your colleagues • Risks/Benefits for your Bosses • Checklists for your Administrators • Next Steps for everyone

5. Game Plan: Why the slides are so “dense”… • The presentation is meant to serve as a take-away • We’ll be moving fast & skimming • You can review the added details later • That way the presentation is still useful without presenter

6. Overview of Situation… • It's 2010 - SharePoint 2007 has been out for a full 3 years; • Even late adopters now have a fully-mature SharePoint 2007 environment; • "Fully-Mature" is code for "everyone's finally using it"; • "Fully-Mature" is code for "at or beyond limits of original design"; • "Fully-Mature" is code for "daily complaints about performance"; • "Fully-Mature" is code for "not putting another dime into it"; • This presentation will cover some "low hanging fruit" which can significantly improve your SP2007 performance, at least long enough to let you deploy your SP2010 farm to new, fast hardware, migrate content and let your "fully-mature" users into the new environment… • (note: "fully-mature users" is also code for "now they take SharePoint for granted, ignore all your warnings about what NOT to do in your farm, turn on versioning 20 levels deep, use WiFi connections 300 ft from the nearest WAP, upload 100 MB files which have NO value to anyone but themselves, install web parts and javascript they KNOW they're NOT supposed to, coerce your developers into "enhancing" your farm WITHOUT your knowledge… and still complain about YOUR farm"… )

7. What can you do?... • We can't talk about everything you could do… • We can talk about a few "must-haves"… • Techniques that are easy to implement • Techniques that apply to all SharePoint environments • Techniques with high benefit-to-risk ratios

8. Tip: Be Methodical in evaluating each improvement… • Devise repeatable tests with: • large & small files (emphasize most common file sizes expected) • varying file types (different file types are affected differently) • Varying operations (typically Page Loads, Downloads, Uploads, Edit/Saves) • short & long distances from The Farm • wide & narrow bandwidth to The Farm • Re-Test again & again as “stack on” each enhancement

10. Must-Have:IIS (HTTP) Compression… Definition: "The seamless, on-the-fly compression, transmission and decompression of file-based content from web server to web browser"; • It's NOT "IIS" Compression; • It IS (more accurately) "HTTP" compression; • Works on outbound (downloads) content delivery • Works with individual files (e.g. htm, aspx, js, css,) comprising a request • Disabled OOTB, but enabled per SP web app during provisioning

11. Explanation: HTTP Compression…

12. Free Tool: Fiddler lets you see HTTP compression… http://www.fiddlertool.com

13. Risks/Benefits:HTTP Compression… • Low Risks: • OOTB in SP2007; • Single Metabase.xml file requires editing • Manual editing possible & straightforward • Readily/Easily adjusted or even reverted • Some ZIP-related oddities; fix available • Requires WFEs which are below 80% Avg CPU (typical) • High Benefits: • Outbound content reductions from 15% to 95% !

14. Checklist: HTTP Compression… C:\windows\system32\inetsrv\metabase.xml: • SAMPLE EDITS: • … • <IIsCompressionSchemeLocation ="/LM/W3SVC/Filters/Compression/gzip" • HcCompressionDll="%windir%\system32\inetsrv\gzip.dll" • HcCreateFlags="1" • HcDoDynamicCompression="TRUE" • HcDoOnDemandCompression="TRUE" • HcDoStaticCompression="TRUE" • HcDynamicCompressionLevel="9" • HcFileExtensions="htm • html • txt • js • css • htc • doc • docx • xls • xlsx • ppt • pptx • pdf" • HcOnDemandCompLevel="9" • HcPriority="1" • HcScriptFileExtensions="asp • aspx • asmx • dll • exe"> • </IIsCompressionScheme> • … PRE-EDIT STEPS: _ OPEN Start > Administrative Tools > IIS Manager _ RMC on Local Server Icon > Properties _ CHECK checkbox property “Enable Direct Metabase Edit” EDIT STEPS: _ LAUNCH cmd.exe > IISRESET /stop _ OPEN Windows Explorer _ NAVIGATE to C:\Windows\System32\inetsrv\ _ LOCATE Metabase.xml file _ DUPLICATE Metabase.xml file > Metabase_20100318.xml _ EDIT Metabase.xml _ SEARCH for text “IISCompressionScheme” (2 elements – gzip & deflate) _ EDIT <IISCompressionScheme> elements (so both match) _ CHANGE Compression Levels to "9" _ INCLUDE all the static & dynamic content YOU have! POST-EDIT STEPS: _ SAVE & CLOSE Metabase.xml _ RESTART IIS via cmd.exe > IISRESET /start _ RE-TEST performance tests &record results _ CHECK with Users (ultimate judges of success)

15. Results: HTTP Compression… • TIP: Test new file formats by batch-ZIP-ing a bunch of them, and do some stats…

16. Thought… • Why does HTTP compression only work on outbound traffic? • (save for discussion…)

18. Uber-Topic: SharePoint (ASP.NET) Web Caching… Definition: Web Caching – The temporary storing (typically in WFE's RAM or VRAM) and retrieving of already-generated HTTP responses or portions of responses. This is MUCH (10x) faster than re-requesting the same page or portions of pages (resources) all the way back to the Database Server…

19. Explanation: SP2007 Web Caching… FAST  SLOW  Output (page) Caching Output (page) Caching Output (page) Caching Browser (private) Caching VRAM BLOB Caching RAM Object Caching Output Caching CHALLENGE: caching is going on everywhere…

20. Confusion: So much caching-related lingo… How Mark makes sense of it all: Caching Data Caching Web Caching Browser-side ("client-side") Caching Server-side Caching Output (Page) Caching Object (Control) Caching BLOB (Resource) Caching Appliance-side Caching (aka "edge" caching)

21. Table: Summarizing SP2007 Web Caching…

22. Type: Browser("private") Caching… • IE7 has built-in "private" caching • Limit is 1024 MB (1 GB)  • Represents the very fastest (shortest) caching • Operation can be hard to administratively control and optimize…

23. Must-Have: Custom cache-control HTTP Header… Browser cache usage can be improved for static websites via custom HTTP response headers; “post-check” is cool trick to load “old” page instantly from browser cache, then update behind-the-scenes as needed…

24. Risks/Benefits: Custom cache-control HTTP Header… • Low Risk • Leverages OOTB caching in IE7 IE8 • Readily/Easily adjusted via IIS Manager at web app • Tends to benefit mostly-static web pages/sites • High Benefits • Fastest possible caching scenario • Significant reduction of Page Load times • Clever leveraging of “effective Page Load” phenomena(“perception is reality…”)

25. Checklist:Custom cache-control HTTP Header… SCHEDULE the following tasks for next maintenance window __ RDC into each WFE in The Farm with admin-level privileges __ PERFORM IISRESET /stop /noforce __ OPEN IIS Manager __ FOR EACH (web application in SharePoint farm) __ RMC > Properties > HTTP Headers __ ADD new custom header: “cache-control: max-age=3600, post-check” __ RESTART IIS via IISRESET /start __ RE-TEST Performance; record results; compare to previous baseline __ EXPERIMENT with max-age (vary between 1 min and 1 hour) __ CHECK with Users (ultimate judges of success)

27. Type: Server-side (WFE) Caching… • Output or "Page" Caching (think "pages & web parts") • BLOB or "Disk" Caching (think "resources/external files") • Object Caching (think "page fragment/control") • All exist on each WFE • All use either WFE RAM or VRAM (disk) • Some share the SAME server resources (RAM/VRAM)! • Each type of caching serves a specific purpose… • Not all purposes apply to all environments • => Not all caching will benefit all situations

28. Must-Have: BLOB (disk) Caching… • AKA "disk-based" caching • Content really is stored on disk instead of RAM • "BLOBS" include all file content typically stored in and retrieved from the database (e.g. js, css, gif, jpg, etc); • Think "resources/files that are also part of a requested page“ • Typically 20-60 resources for SharePoint page ! (significant) • Part of ASP.NET (built right into native IIS performance counters) • Disabled by default in SP2007; • Enabled on per-web application basis (easy to manage)

29. Explanation: BLOB (disk) Caching… Edit each web app's web.config file:

30. Risks/Benefits: BLOB (disk) Caching… • Low Risk • OOTB in SP2007; • Single line of web.config file; • Highly configurable wrt location, size, selection; • Location off of WFE further improves performance; • Readily/Easily adjusted or even disabled; • Managed at the Web Application-level (*whew*) • High Benefits • Reduction of Browser Load times ranging from ½ to 1/7th of un-cached load times

31. Checklist: BLOB (disk) Caching… SCHEDULE the following tasks for next maintenance window __ PRE-TEST performance of BLOB retrieval using Fiddler 2 (record download times of largest files) __ RDC into each WFE in The Farm with admin-level privileges __ PERFORM IISRESET /stop /noforce __ EXPLORE C:\inetpub\wwwroot\wss\virtualdirectories\ __ NAVIGATE into each web application sub-folder __ FOR-EACH (web application) DO LOCATE the web.config file COPY web.config file to web.config_2010.03.18 (best practice); OPEN web.config file in edit mode FIND "<Blobcache" text string => should yield only a single element EDIT <Blobkcache string as desired - particularly location, size & file types SAVE web.config file __ PERFORM IISRESET /start __ RE-TEST performance __ CHECK with Users (ultimate judges of success)

33. Must-Have: Object Caching… • Located in the VRAM of each WFE (shares resources with BLOB Cache) • Caches • Site Navigation Structure • Content in the different fields of a document library item or a list item • Published pages & resources • Draft pages • Manifested to Users as good/bad navigation, queries and edits/saves • Enabled by default • Controls a single site collection at a time (challenging for some farms) • Defaults to 100 MB; ranges from 100 MB to 300 MB max per site collection • Shares both same memory space & same process as BLOB Cache • Need to be careful about over-allocating memory for this cache • Need to be REALLY careful about under-allocating memory for this cache

34. Risks/Benefits: Object Caching… • Moderate Risk • OOTB & on-by-default in SP2007; • Only need to adjust Object Cache Size (MB); • Readily/Easily adjusted via Site Collection Admin Page • DO need to be careful about balancing shared server resources • OVER-allocating immediately & negatively impacts performance • UNDER-allocating can create "cache thrashing" – VERY BAD! • Annoying to manage for large site collection counts • Tuning techniques range from complex to simplistic (go for simple) • Often tuned manually, and only on high-use site collections ! • Moderate Benefits • Already working with default of 100 MB (per site collection) • Varying reduction of resource load times

35. Checklist:Object Caching… SCHEDULE the following tasks for next maintenance window __ START with all Site Collections all remaining at default Object Cache Size = 100 MB __ MONITOR (via PerfMonitor Counter Log ) SharePoint Publishing Cache/Number of Compactions low/zero compactions and good edit/save feedback from users = > probably leave as-is high compactions and poor edit/save feedback from users => try increasing in 25 MB increments NEVER reduce without having valid reason (thrashing horror stories abound) __ DETERMINE (various means) all "high latency" site collections FOR-EACH (site collection) __ BROWSE to Site Actions > All Site Settings > Site Collection Admin > Site Collection Object Cache __ CONFIRM current Object Cache Setting (100 MB default) __ INCREASE value by 50 MB __ SAVE & RESTART SERVER !! (or previous counts will not be reset) __ RE-TEST performance (particularly navigation and edits/saves) __ CHECK with Users (Ultimate judges of success) OPTIONAL – leverage Performance Counter to quantify optimal settings (overkill compared to other effort)

37. Must-Have: Increased Browser-Server Connections… Both IE7 and IE8 OOTB have fewer max server connections than they could have… IE7 defaults to 2 server connections IE8 defaults to 6 server connections Microsoft provides a fix to increase this value to 10: http://support.microsoft.com/default.aspx/kb/282402?p=1

38. Risks/Benefits: Increased Browser-Server Connections… Low Risk: Enhancement comes directly from Microsoft Can be implemented by individual users or via SMS Can be tuned/adjusted to optimize LAN bandwidth Daring to go above 10 connections => perception of DNS ! High Benefits: Increased # concurrent downloads => faster page loads IE7 (2 connections): 10 sec page loads IE7 (10 connections): 3 sec page loads

39. Checklist: Increased Browser-Server Connections… __ BROWSE to http://support.microsoft.com/default.aspx/kb/282402?p=1 __ CLICK “Fix it” button __ RE-TEST Performance against current baseline

41. Must-Have: Ethernet (not WiFi) Connections Connection Type is not sexy, but in many environments this single apparently insignificant decision can eclipse all other attempts to improve SP2007 performance combined. Example: Your Ethernet connection (ideally) provides 25 Mb/s of download bandwidth; Your pre-tuning performance gave 20 secs to download a 20 MB file; Your Tunings improve this to 10 secs (a whopping 50% gain); BUT… 4. Your users like the convenience of WiFi, so they ignore your best practices; 5. Your users set up 75 ft from the nearest WAP, because they like the view; 6. At this location, their Effective Bandwidth is now 5 Mb/s for downloads; SO… 7. Downloading that same file now takes 50 secs, AND USERS COMPLAIN.

43. Also Investigate: Proxy Server Caching… • Most proxy servers these days also provide static web content (output) caching – investigate this if your WFEs are memory-limited…

44. Also Investigate: Load Balancer Caching… • Many current load balancers also provide static web content (output) caching. Investigate this as well if your WFEs are memory limited…

45. Nice-to-Have: Riverbed’s Steelhead CC Appliance & Client… • Does Caching & Compression like other technologies • Differentiator is that it handles BOTH outbound AND inbound traffic

46. Nice-to-Have: Aptimize ISAPI Filter… • Bolts-on to IIS • Configured per web app (handy) • Post-processes HTTP responses • Packs js,css • Minifies & gzips resources • Sprite-ifies images • Improves caching • Remote Page Times improved 65% • Local Page Times improved 37% • Used on SharePoint.microsoft.com

48. Next Steps… __ SKETCH out your own environment's Content Delivery Chain, from Database to Browser __ IDENTIFY all possible points of caching & compression __ MEASURE baseline performance metrics (Content Download Rate, Content Upload Rate, Page Load Time, Edit/Save Time, others) __ ASSESS current IIS (HTTP) Compression settings on each WFE __ INVENTORY all commonly-used file types (e.g. doc, docx, xls, xlsx, ppt, pptx, pdf, etc) __ PREPARE changes/modifications needed to the HTTP Compression settings __ INVENTORY Disk Caching Settings on all WFEs __ INVENTORY BLOB Caching Settings on all WFEs __ INVENTORY Object Cache in “key” Site Collections (ugh) __ IMPLEMENT improvements ONE AT A TIME __ RE-TEST/MEASURE performance improvements & compare to baseline __ CHECK WITH USERS (ultimate judges of success)

49. Conclusion… • HOW TO REQUEST A COPY OF THIS PRESENTATION: • POSTED on CSPUG website • EMAIL mark.vogt@vogtland.ws or mark_vogt@hotmail.com • VISIT MARK’S BLOG: • www.markvogt.us or www.thewiseconsultant.us • OTHER AVAILABLE PRESENTATIONS: • PowerShell v2.0 for SharePoint 2007 Reporting, Policing & Administration • The Perfect SharePoint Server Farm • Governing SharePoint…with SharePoint • SharePoint-enabled Project Management Methodology

50. Discussion… Q: WHAT experiences have others had with these techniques? Q: WHAT other performance-relevant best practices can others share? Q: WHAT other inbound technologies have others had success with? Q: WHAT techniques have others developed for quickly/programmatically inventorying various caching settings? Q: WHAT does Microsoft literature mean when it states that BLOB Caching 100 MB is "per site collection"? What are the best practices for environments with thousands of low-use site collections and users which demand good performance? Q: What happens when you circumvent Central Admin, browse directly to the Output Cache URL, and set it. Can you get Output Caching for NON-publishing websites?