Parallel and Distributed Processing CSE 8380

1. Parallel and Distributed ProcessingCSE 8380 February 1 2005 Session 6

2. Contents • Performance Evaluation • Grosch’s Law • Moore’s Law • Von Neumann’s Bottlneck • Parallelism • Speedup • Amdahl’s Law • The Gustafson-Barsis Law • Benchmarks

3. Grosch’s Law (1960s) • “To sell a computer for twice as much, it must be four times as fast” • Vendors skip small speed improvements in favor of waiting for large ones • Buyers of expensive machines would wait for a twofold improvement in performance for the same price.

4. Moore’s Law • Gordon Moore (cofounder of Intel) • Processor performance would double every 18 months • This prediction has held for several decades • Unlikely that single-processor performance continues to increase indefinitely

5. Von Neumann’s bottleneck • Great mathematician of the 1940s and 1950s • Single control unit connecting a memory to a processing unit • Instructions and data are fetched one at a time from memory and fed to processing unit • Speed is limited by the rate at which instructions and data are transferred from memory to the processing unit.

6. Problem Assume that a switching component such as a transistor can switch in zero time. We propose to construct a disk-shaped computer chip with such a component. The only limitation is the time it takes to send electronic signals from one edge of the chip to the other. Make the simplifying assumption that electronic signals travel 300,000 kilometers per second. What must be the diameter of a round chip so that it can switch 109 times per second? What would the diameter be if the switching requirements were 1012 time per second?

7. Parallelism • Multiple CPUs • Within the CPU • One Pipeline • Multiple pipelines

8. Superscalar Parallelism Scheduling

9. Past Trends in Parallel Architecture (inside the box) • Completely custom designed components (processors, memory, interconnects, I/O) • Longer R&D time (2-3 years) • Expensive systems • Quickly becoming outdated • Bankrupt companies!!

10. New Trends in Parallel Architecture (outside the box) • Advances in commodity processors and network technology • Network of PCs and workstations connected via LAN or WAN forms a Parallel System • Network Computing • Compete favorably (cost/performance) • Utilize unused cycles of systems sitting idle

11. Speedup • S = Speed(new) / Speed(old) • S = Work/time(new) / Work/time(old) • S = time(old) / time(new) • S = time(before improvement) / time(after improvement)

12. Speedup • Time (one CPU): T(1) • Time (n CPUs): T(n) • Speedup: S • S = T(1)/T(n)

13. Amdahl’s Law The performance improvement to be gained from using some faster mode of execution is limited by the fraction of the time the faster mode can be used

14. Example 20 hours B A must walk 200 miles Walk 4 miles /hour Bike 10 miles / hour Car-1 50 miles / hour Car-2 120 miles / hour Car-3 600 miles /hour

15. Example 20 hours B A must walk 200 miles Walk 4 miles /hour 50 + 20 = 70 hours S = 1 Bike 10 miles / hour 20 + 20 = 40 hours S = 1.8 Car-1 50 miles / hour 4 + 20 = 24 hours S = 2.9 Car-2 120 miles / hour 1.67 + 20 = 21.67 hours S = 3.2 Car-3 600 miles /hour 0.33 + 20 = 20.33 hours S = 3.4

16. Amdahl’s Law (1967) •  : The fraction of the program that is naturally serial • (1- ): The fraction of the program that is naturally parallel

17. S = T(1)/T(N) T(1)(1-  ) T(N) = T(1) + N 1 N S = = (1-  )  + N + (1-  ) N

18. Amdahl’s Law

19. Gustafson-Barsis Law N &  are not independent from each other a : The fraction of the program that is naturally serial T(N) = 1 T(1) = a + (1- a ) N S = N – (N-1) a

20. Gustafson-Barsis Law

22. Distributed Computing Performance • Single Program Performance • Multiple Program Performance

24. Benchmark Performance • Serial Benchmarks • Parallel Benchmarks • PERFECT Benchmarks • NAS Kernel • The SLALOM • The Golden Bell Prize • WebSTONE for the Web • Performance Comparisons