1 / 37

Capacity Planning for LAMP Architectures

Capacity Planning for LAMP Architectures. Web Builder 2.0 Las Vegas. John Allspaw Manager, Operations Flickr.com. Capacity Planning for LAMP Architectures. Capacity planning: “ the ability to make snap decisions to spend millions of dollars with not enough information”

aurora
Télécharger la présentation

Capacity Planning for LAMP Architectures

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Capacity Planning for LAMP Architectures Web Builder 2.0 Las Vegas John Allspaw Manager, Operations Flickr.com

  2. Capacity Planning for LAMP Architectures Capacity planning: “the ability to make snap decisions to spend millions of dollars with not enough information” - Kevin Murphy

  3. Capacity Planning for LAMP Architectures It is NOT: • Performance tuning • Tips and tricks to be “scalable”

  4. Capacity Planning for LAMP Architectures It IS: • What comes after you’ve made it all “scalable” • Making sure that you have enough equipment to handle gradual and bursty traffic

  5. Capacity Planning for LAMP Architectures Questions to answer

  6. Capacity Planning for LAMP Architectures How many of each server “class” should you add as you grow ? Hint: Don’t add too much (too much $$! Ahh!) Hint: Don’t add too little (too much traffic! Ahh!)

  7. Capacity Planning for LAMP Architectures How to make it easier to predict the future* ? How to make it easier to justify those predictions ? How to make it easier to predict the future…in the future ?! *You can’t predict the future, but you can try.

  8. Capacity Planning for LAMP Architectures The OLD way of doing things was easier • A.k.a. “web1.0” • Small number of content producers • Control over the content • Capacity was dictated by the demand for that static/small content • Even bbs/communities/ecommerce had relatively predictable growth

  9. Capacity Planning for LAMP Architectures Today’s way of doing things is harder/fun • No control over content (users have control) • No control over usage (users have control)

  10. Capacity Planning for LAMP Architectures Today’s way of doing things is harder/fun • Network effect, nonlinear growth (more users/content/contacts/activity mean >> usage) • Event-related growth (press, news event, social trend, etc. can affect usage and content) Example: London bombing, tsunami, holidays, etc.

  11. Capacity Planning for LAMP Architectures Considerations for “social” applications • User behavior should guide you with defining capacity metrics (not just server stats) • Usage can accelerate, not just grow

  12. Capacity Planning for LAMP Architectures How we do it at Flickr

  13. Capacity Planning for LAMP Architectures Gathering Usage • Application-level information (users, photos, activity, etc.) • Server-level information (cpu, disk I/O, memory, etc.) • We tie the two together

  14. Capacity Planning for LAMP Architectures BEFORE we start collecting server stats • What resources are peak-driven ? (concurrent use) • Ex: photo processed/sec, pages/sec, images/sec, db qps • What resources are permanently consumable ? • Ex: database space, storage (GB/day) etc.

  15. Capacity Planning for LAMP Architectures BEFORE we start collecting server stats • What is an average user:consumption ratio ? (example: user: photo) • What is the high and low of ratios ? • Is the average ratio changing over time ?

  16. Capacity Planning for LAMP Architectures Non-linear growth

  17. Capacity Planning for LAMP Architectures Non-linear growth

  18. Capacity Planning for LAMP Architectures Linear relationships, though

  19. Capacity Planning for LAMP Architectures Server and Network statistics • Ganglia - (we love ganglia!) • Multicast-y goodness • SUPER simple to make a graph from any stat • Clustering • Other custom rrdtool-based stuff • MRTG

  20. Capacity Planning for LAMP Architectures Photos uploaded/processed/min Avg processing time per minute Avg CPU per minute

  21. Capacity Planning for LAMP Architectures Gather and record statistics • Accept the ‘observer effect’ (it’s worth it) • Aggregate your stats across clusters • Stacked graphs • Totals and averages

  22. Capacity Planning for LAMP Architectures Squid client requests (24 hours) (Y axis is req/sec)

  23. Capacity Planning for LAMP Architectures Squid LRU reference age Over 24 hours • Y axis is “days” • So peak has 3.6hours

  24. Capacity Planning for LAMP Architectures Find the ceiling of each class/function/server • Maximum allowable somethings • MySQL : queries/sec before slave lag sets in • Apache/php : page requests/sec before total meltdown • Squid/memcached : cache churn rate, request rate • Storage : disk I/O utilization, storage limit(!) • Etc.

  25. Capacity Planning for LAMP Architectures Forget benchmarking, use real load • Make sure you have a easy mechanism to take servers in and out of production • Pull machines from a balanced pool during medium-level traffic (very carefully) • Watch and record

  26. Capacity Planning for LAMP Architectures Build the infrastructure to make it EASY to measure Obvious things to help this: • Load balancing • Network segmentation • Carve up functions into clusters • Don’t let a machine do more than one primary thing (if you can help it) this isn’t for performance! If it makes it faster/better, then bonus!

  27. Capacity Planning for LAMP Architectures For graphs you don’t have raw data for GraphClick http://www.arizona-software.ch/applications/graphclick/en/ - graph digitizer package - $8 US - you pick points on a calibrated image, it spits out tabular data

  28. Capacity Planning for LAMP Architectures

  29. Capacity Planning for LAMP Architectures Once you have: • Time history of metrics • “Ideal” peak levels (ceiling) Then you can: 3. Predict the future!*

  30. Capacity Planning for LAMP Architectures Example: Photo Processing

  31. Capacity Planning for LAMP Architectures Photos uploaded/processed/min Avg processing time per minute Avg CPU per minute

  32. Capacity Planning for LAMP Architectures

  33. Capacity Planning for LAMP Architectures Dirty linear math 25% CPU @40 photos/min 40% CPU @60 photos/min So….take a “ceiling”: 75% CPU @112 photos/min = 6720 photos/hour (but double-check the process time)

  34. Capacity Planning for LAMP Architectures Conclusion

  35. Capacity Planning for LAMP Architectures • Know your machines and their limits • Measure how the site is being used with application-level stats • Tie real-world observations to server stats

  36. Capacity Planning for LAMP Architectures Some Flickr statistics • 300M photos, 4 or 5 different sizes • Keep ~25M images in cache at any time, ~1M from RAM • 2B MySQL queries/day • 21k req/sec to memcached • 1.2 PT raw disk storage

More Related