Overview of Parallel Architecture

1. Overview of Parallel Architecture Yulia Newton CS 147, Fall 2009 SJSU

2. What is it? • “Parallel processing is the ability of an entity to carry out multiple operations or tasks simultaneously. The term is used in the contexts of both human cognition and machine computation.” - Wikipedia

3. Humans • Human body is a parallel processing architecture • Human body is the most complex machine known to man • We perform parallel processing all the time

4. Why? • We want to do MORE • We want to do it FASTER

5. Limitations of Single Processor • Physical • Heat • Electromagnetic interference • Economic • Cost

6. Parallel Computing • Divide bigger tasks into smaller tasks • Reminds you Divide and Conquer concept? • Perform calculations of those tasks simultaneously (in parallel)

7. With Single CPU • Parallel processing is possible with single-CPU, single-core computers • Requires very sophisticated software called distributed processing software • Most parallel computing is done with multiple CPU’s

8. Speedup from parallelization • More processors = faster computing? • Speed-up from parallelization is not linear Can do single task in a unit of time Can do twice as many tasks in same the amount of time Can do four times as many tasks in the same amount of time

9. Speedup • Definition: Speedup is how much a parallel algorithm is faster than a corresponding sequential algorithm.

10. Gene Myron Amdahl • Norwegian American computer architect and hi-tech entrepreneur

11. Amdahl's Law • Formulated in 1960s • Gives potential speed-up on parallel computing system Small portion of components/elements that cannot be parallelized will limit speed-up possible from parallelizable components/elements.

12. Amdahl's Law (cont’d) where : • S is the overall speed-up of the system • P is the fraction that is parallelizable

13. Amdahl's Law Alternative Form • where • f – fraction of the program that is enhanced • s – speedup of the enhanced portion

14. Amdahl's Law (cont’d)

15. Amdahl's Law Example Program • Program has four parts Code 1 Part 1 = 11% Loop 1 Part 2 = 18% Loop 2 Part 3 = 23% Code 2 Part 4 = 48%

16. Amdahl's Law Example (cont’d) • Let’s say: • P1 is not sped up, so S1 = 1 or 100% • P2 is sped up 5×, so S2 = 500% • P3 is sped up 20×, so S3 = 2000% • P4 is sped up 1.6×, so S4 = 160%

17. Amdahl's Law Example (cont’d) • Using P1/S1 + P2/S2 + P3/S3 + P4/S4:

18. Amdahl's Law Example (cont’d) • 0.4575 is a little less than ½ of the original running time, which is 1 • Overall speed boost is 1 / 0.4575 = 2.186 or a little more than double the original speed • Notice how the 20× and 5× speedup don't have much effect on the overall speed boost and running time when 11% is not sped up, and 48% is sped up by 1.6×

19. Amdahl's Law Limitations • Based on fixed work load or fixed problem size (strong or hard scaling) • Implies that machine size is irrelevant

20. John L. Gustafson • American computer scientist and businessman

21. Gustafson's Law • Formulated in 1988 • Closely related to Amdahl’s Law • Addresses the shortcomings of Amdahl's law, which cannot scale to match availability of computing power as the machine size increases

22. Gustafson's Law (cont’d) • where P is the number of processors, S is the speedup, and α the non-parallelizable part of the process

23. Gustafson's Law (cont’d) • Proposes a fixed time concept which leads to scaled speed up for larger problem sizes • Uses weak or soft scaling

24. Gustafson's Law Limitations • Some problems do not have fundamentally larger datasets • Example: processing one data point per world citizen gets larger at only a few percent per year • Nonlinear algorithms may make it hard to take advantage of parallelism "exposed" by Gustafson's law

25. Parallel Architectures • Superscalar • VLIW • Vector Processors • Interconnection Networks • Shared Memory Multiprocessors • Distributed Computing • Dataflow Computing • Neural Networks • Systolic Arrays

26. Superscalar Architecture • Implements instruction-level parallelism (multiple instructions are executed simultaneously in each cycle) • Net effect is the same as pipelining • Additional hardware is required • Contains specialized instruction fetch unit (retrieves multiple instructions simultaneously from memory) • Relies on both hardware and compiler

27. Superscalar Architecture • Analogous to adding a lane to a highway – more physical resources are required but in the end more cars can pass through

28. Superscalar Architecture • Examples: • Pentium x86 processors • Intel i960CA • AMD 29000-series

29. VLIW Architecture • Stands for Very Long Instruction Word • Similar to Superscalar architecture • Relies entirely on compiler • Pack independent instructions into one long instruction • Bigger size of compiled code

30. VLIW Architecture • Examples: • Intel’s Itanium, IA-64 • Floating Point Systems' FPS164

31. Vector Processors • Supercomputers are vector processor systems • Specialized, heavily pipelined processors • Efficient operations on entire vectors and matrices

32. Vector Processors • Heavily pipelined so that arithmetic operations can be overlapped

33. Vector Processors (example) • Instructions on a traditional processor: For I = 0 to VectorLength V3[i] = V1[i] + V2[i] • Instructions on a vector processor: LDV V1, R1 LDV V2, R2 ADDV R3, R1, R2 STV R3, V3

34. Vector Processors • Applications: • Weather forecasting • Medical diagnoses systems • Image processing

35. Vector Processors • Examples: • Cray series of cupercomputers • Xtrillion 3.0

36. Interconnection Networks • MIMD (Multiple instruction stream, multiple data stream) type architecture • Each processor has its own memory • Processors are allowed to access other processors’ memory via network

37. Interconnection Networks • Can be: • Dynamic – allow paths between two components to change from one communication to another • Static – do not allow a change of path between communications

38. Interconnection Networks • Can be: • Non-blocking – allow new connections in the presence of another simultaneous connections • Blocking – do not allow simultaneous connections

39. Interconnection Networks • Sample interconnection networks:

40. Shared Memory Architecture • MIMD (Multiple instruction stream, multiple data stream) type architecture • Single memory pool accessed by all processors • Can be categorized by how the memory access is performed: • UMA (Uniform Memory Access) – all processors have equal access to entire memory • NUMA (Non-Uniform Memory Access) – each processor has access to its own piece of memory

41. Shared Memory Architecture • Diagram of a Shared Memory system:

42. Distributed Computing • Very loosely coupled multicomputer system, connected by buses or through a network • Main advantage is cost • Main disadvantage is slow communication due to the physical distance between computing units

43. Distributed Computing • Example: • SETI (Search for Extra Terrestrial Intelligence) group at UC Berkley analyzes data from radio-telescopes • To help this project PC users can install SETI screen saver on their home computer • Data will be analyzed during the processor’s idle time • Half a million years of CPU time accumulated in about 18 months!

44. Data Flow Computing • Von Neumann machines exhibit sequential control flow; data and instructions are segregated • Dataflow computing – control of the program is directly tied to the data • Order of the instructions does not matter • Each instruction is considered a separate process • Tokens are executed rather than instructions

45. Data Flow Computing • Data Flow Graph for y=(c+3)*f

46. Data Flow Computing • Data flow system diagram:

47. Neural Networks • Good for massively parallel applications, fault tolerance and adapting to changing circumstances • Based on parallel architecture of human brain • Composed of large number of simple processing elements • Trainable

48. Neural Networks • Model human brain • Perceptron is the simplest example

49. Neural Networks • Applications: • Quality control • Financial and economic forecasting • Oil and gas exploration • Speech and pattern recognition • Health care cost reduction • Bankruptcy prediction

50. Systolic Arrays • Networks of processing elements that rhythmically compute data by circulating it through the system • Variations of SIMD architecture