1 / 27

System Modeling and Performance Evaluation

System Modeling and Performance Evaluation. Anand Tripathi CSci 5104. Course Objective. The objective of this course is to learn techniques for modeling computing systems for performance evaluation through analytical as well as simulation methods.

ami
Télécharger la présentation

System Modeling and Performance Evaluation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. System Modeling and Performance Evaluation Anand Tripathi CSci 5104

  2. Course Objective • The objective of this course is to learn techniques for modeling computing systems for performance evaluation through analytical as well as simulation methods. • The students would learn how to model computing systems and network communication protocols to evaluate their performance under different operating conditions.  • Analytical techniques to be studied would include operational analysis as well stochastic techniques for queueing networks

  3. Textbooks Queueing Systems Volume 1: Theory by Leonard Kleinrock, John Wiley Quantitative System Performance Computer System Analysis Using Queueing Network ModelsEdward D. Lazowska, John Zahorjan, G. Scott Graham, Kenneth C. Sevcik , John Wiley (This book is now available free on the Internet by the publisher and the authors.) Additionally about 10-12 papers. 3

  4. Course Resources • See course website at • http://www.cselabs.umn.edu/classes/Fall-2011/csci5104/ • This website has: • Lecture notes posted around lecture dates • Lecture schedule and exam schedule • Grading criteria • Link to online book by Lazowska et al • Example programs and reading list papers • CSIM programming guide • Streaming video on UNITE available 10 days after the lecture

  5. Required Background Basic knowledge of probability theory Basic calculus Familiarity with the C programming language Some basic knowledge of computing system organization 5

  6. WORK LOAD Pages of reading/week: 20-25 from textbook ; additional reading may involve reading of about 10-12 research/technical papers during the semester. The course involves about 6 assignments. Assignments would involve programming and problem solving exercises. One project to be performed in group of two students. 6

  7. GRADING Six assignments 50% About 30% concept oriented problems Number of tests/exams: One midterm exam November 3 15% Final exam December 22 25% Exam format:Problem solving Term projects: 10% Proposal due on October 11 Final report due on December 13 7

  8. Major Topics • Operational Analysis Techniques • Topics covered in the book by Lazowska et al, up to Chapter 6 • Queueing theory based on Markov models • Topics from Kleinrock’s book, up to Chapter 4 • System simulation methods • CSIM programming language manual • Other topics from lecture notes

  9. Operational Analysis Techniques • Simple methods based on average values for • Job arrival rates, service times, resource usage demands • Little’s law and Forced Flow Law: • Relationships between queuelengths, arrival rates, and service times • Mean Value Analysis (MVA) techniques • Performance bounds

  10. Queueing models and analysis Stochastic models of queueing systems Basic probability theory Use of Laplace and Z transform methods Poisson processes and distribution, exponential distribution for arrival/service times Markov processes, M/M/1 queues and other variations M/G/1 systems Network of queues (Buzen’s algorithm) 10

  11. Simulation techniques Event driven simulation techniques Introduction to CSIM programming language Analysis of simulation data and statistics. Methods for random number generation in simulators. The students would develop simulation models using CSIM Other models such NS-2 network simulator may be used in projects 11

  12. Workload characterization Web server load characterization Various SPEC benchmarks Cloud workload models Yahoo Cloud Serving Benchmark (YCSB); Program execution profiling and monitoring tools such as perfmon on Linux 12

  13. Case studies • Applications of performance analysis methods would be studied using several case studies, which would include modeling and performance evaluation of  Web servers,  multi-stage/multi-tier execution models in cluster-based systems, application scheduling and task placement in cluster-based  and Cloud based hosting platforms, and communication protocols in network systems.

  14. These methods can be used at different stages and phases: Analysis during system design phase To show that a given design would meet certain performance goals System acquisition System performance tuning and upgrades Use of system modeling methods

  15. System modeling at design time • During the design of a new system to evaluate different design choices. • Impact of different parameters and architectural choices • Protocols and algorithm choices • Resource management, coordination • Scheduling mechanisms and choices • Capacity selection for subsystems • Understand how the system would perform

  16. System modeling for acquisition phase • During system acquisition of system, either the vendor or the customer may perform modeling and evaluation to determine the appropriate system configuration for the expected workload and operating conditions.

  17. System modeling for tuning and upgrades • To deal with evolution of workload during the lifetime of a system. The workload characteristics may change or evolve. • Customer population may increase or the job characteristics may change significantly, requiring changes in system configuration or upgrading of service capacities of the component systems.

  18. What is the throughput of the system for a given workload: Number jobs/customers serviced in unit time? What is the response time experienced by the users? What is the turnaround time for a job. What is the utilization of different subsystems? Which subsystem is most utilized? Bottlenecks? What is the number of jobs waiting for service at different subsystems? What is time of residency of a job in a specific subsystem? Performance Measures

  19. One can determine various measures for the performance metrics: Bounds for their values Upperbound and lowerbound (range) Mean, variance, median, mode values Probability distributions for the measures Performance Measures

  20. Many times the designers or the system administrators are interested in answering the “What if ..” questions. What if the workload changes in certain ways? What if a higher capacity disk is added to the system? What if more memory is added to the system? What if another processor is added to the system? What is the impact of specific parameter in the design of a system or protocol? Parameterized Models

  21. Two extremes approaches for performance prediction • Trend exploration based on experience and intuition • Largely dependent on individual experience • Accuracy is suspect • Experimental evaluation of alternatives • Expensive to modify an actual system and evaluate all alternatives. • May not be feasible due to cost of different subsystems needed for experimentations,

  22. System Modeling • A model represents an abstraction for the system. • It only need to capture the important relevant details for the various subsystems. • Only those details at affect the performance related measures. • Model development is a top-down process: • Starting with the high-level system architecture, the subsystems are refined further to the details that influence the measures of interest. • In some cases it may not be possible to acquire all fine details of a subsystem, and only broad measures and characteristics may be utilized. • Some characteristics may need to be measured

  23. Model Development and Use • Model validation • Baseline model for an existing system • Workload measurements • Validate against the measurement data from the operational system • Projection Phase • Model inputs or parameters are modified to reflect the anticipated changes in the workload or the system configuration • Verification • The measurements from the modified system are comapred with those predicted using the model.

  24. A large system can be decomposed as a collection of subsystems . We call them “service stations”. A job in the system requires service different subsystems. A job “visits” different service stations. Closed vs. Open system A closed system has a fixed set of customer population, and the customers continually move through different subsystems. In an open system, customer arrive, get service at different subsystem, and then leave the system. Queueing Network Models

  25. Single Server System Server Queue of Waiting Customers Arriving customers or requests Departing customers or requests A large service facility can be viewed as a network of such basic server-server queueing systems. CSCI 5104 - Tripathi

  26. Queueing Network Models of Computing Systems Queue of Ready-to-run Processes cpu Departing jobs or requests Arriving Jobs or or requests Queue of disk-1 I/O requests disk1 This system has multiple service stations, each with its own queue of requests Queue of disk-2 I/O requests disk2 CSCI 5104 - Tripathi

  27. Queueing Network Model of a Timesharing System User Terminals Queue of Ready-to-run Processes cpu Queue of disk-1 I/O requests disk1 Queue of disk-2 I/O requests disk2 CSCI 5104 - Tripathi

More Related