1 / 24

Parallel Programming Languages

Parallel Programming Languages. Andrew Rau-Chaplin. Sources. D. Skillicorn, D. Talia, “Models and Languages for Parallel Computation”, ACM Comp. Surveys. Warning: This is very much ONE practitioners viewpoint! Little attempt has been made to capture the conventional wisdom. Outline.

dakota
Télécharger la présentation

Parallel Programming Languages

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Parallel Programming Languages Andrew Rau-Chaplin

  2. Sources • D. Skillicorn, D. Talia, “Models and Languages for Parallel Computation”, ACM Comp. Surveys. • Warning: This is very much ONE practitioners viewpoint! Little attempt has been made to capture the conventional wisdom.

  3. Outline • Introduction to parallel programming • Example Languages • Message Passing in MPI • Data parallel programming in *Lisp • Shared address space programming in • OpenMP • CILK

  4. Historically • Supercomputers • Highly structured numerical programs • Parallelization of loops • Multicomputers • Each machine had its languages/compilers/libraries optimized for its architecture • Parallel computing for REAL computer scientist, “Parallel programming is tough, but worth it” •  Mostly numerical/scientific applications written using Fortran and parallel numerical libraries •  Little other parallel software was written!

  5. Needed • Parallel programming abstractions that where • Easy – provide help managing programming complexity • But general! • Portable – across machines • But efficient!

  6. Solution: Yet another layer of abstraction! Application Software System Software Parallel Model/ Language Systolic Arrays SIMD Generic Parallel Architecture Message Passing Dataflow Shared Memory

  7. CAD Database Scientific modeling Parallel applications Multipr ogramming Shar ed Message Data Pr ogramming models addr ess passing parallel Compilation Communication abstraction or library User/system boundary Operating systems support Har dwar e/softwar e boundary Communication har dwar e Physical communication medium Layered Perspective [Language = Library = Model]

  8. Programming Model • Conceptualization of the machine that programmer uses in coding applications • How parts cooperate and coordinate their activities • Specifies communication and synchronization operations • Multiprogramming • no communication or synch. at program level • Shared address space • like bulletin board • Message passing • like letters or phone calls, explicit point to point • Data parallel: • more regimented, global actions on data • Implemented with shared address space or message passing

  9. What does parallelism add? • Decomposition • How is the work divided into distinct parallel threads? • Mapping • Which thread should be executed on which processor? • Communication • How is non-local data acquired? • Synchronization • When must threads know that they have reached a common state?

  10. Skillicorn’s Wish list • What properties should a good model of parallel computation have? • Note: desired properties may be conflicting • Themes • What does the programming model handle for the programmer? • How abstract can the model be and still realize efficient programs? • Six Desirable Features

  11. 1) Easy to program • Should conceal as much detail as possible • Example of 100 proc., each with 5 threads, each thread potential communicated with any other = 5002 possible communication states! • Hide: Decomposition, Mapping, Communications, and Synchronization • As much as possible, rely on translation process to produce exact structure of parallel program

  12. 2) Software development methodology • Firm semantic foundation to permit reliable transformation • Issues: • Correctness • Efficiency • Deadlock free Parallel Model/ Language Parallel Architecture

  13. 3) Architecture-Independent • Should be able to migrate code easily to next generation of an architecture • Short cycle-times • Should be able to migrate code easily from one architecture to another • Need to share code • Even in this space, people are more expensive and harder to maintain than hardware

  14. 4) Easy to understand • For parallel computing to be main stream • Easy to go from sequential  Parallel • Easy to teach • Focus on • easy-to-understand tools with clear, if limited, goals • over, complex ones that may be powerful but are hard to use/master!

  15. 5) Guaranteed performance • Guaranteed performance on a useful variety of real machines • If T(n,p) = c f(n,p) + low order terms • Preserve the Order of the complexity • Keep the constants small • A model that is good (not necessarily great) on a range of architectures is attractive!

  16. 6) Provide Cost Measures • Cost measures are need to drive algorithmic design choices • Estimated execution time • Processor utilization • Development costs • In sequential, • executions times between machines proportional (Machine A is 5 times faster than Machine B) •  Two step model: Optimize algorithmically then code and tune.

  17. 6) Provide Cost Measures cont. • In Parallel, • Not so simple, no two step model • Costs associated with decomposition, Mapping, Communications, and Synchronization may vary independently! •  model must make estimated cost of operations available at design time • Need an accounting scheme or cost model! • Example: How should an algorithm trade-off communication vs. local computation?

  18. Summary: Desired Features • Often contradictory • Some features more realistic on some architectures • Room for more than one Language/Model!

  19. Six Classification of Parallel Models More Abstract,Less Efficient (?) 1) Nothing Explicit, Parallelism Implicit 2) Parallelism Explicit, Decomposition Implicit 3) Decomposition Explicit, Mapping Implicit 4) Mapping Explicit, Communications Implicit 5) Communications Explicit, Synchronization Implicit 6) Everything Explicit Less Abstract,More Efficient (?)

  20. Within Each Classification • Dynamic Structure • Allows dynamic thread creation • Unable to restrict communications • May overrun communication capacity • Static Structure • No dynamic thread creation • May overrun communication capacity, cut • Static structure supports cost models for prediction of communication • Static and Communication Limited Structure • No dynamic thread creation • Can guarantee performance by limiting frequency and size of communications

  21. 1 4 5 2 Languages Cilk Libraries 6 *Lisp Where should OpenMP go ??? 3 Models

  22. Recent Languages/systems • Cilk • http://supertech.csail.mit.edu/cilk/ • http://www.cilk.com/ • http://software.intel.com/en-us/intel-cilk-plus • MapReduce • http://labs.google.com/papers/mapreduce.html • http://hadoop.apache.org/

  23. Recent Languages • GPUs: OpenCL & CUDA • http://www.khronos.org/opencl/ • http://www.nvidia.com/object/cuda_home.html • https://developer.nvidia.com/category/zone/cuda-zone • Grid Programming • http://www.globus.org/ • http://www.cct.lsu.edu/~gallen/Reports/GridProgrammingPrimer.pdf

  24. Recent Languages • Cloud Computing • http://aws.amazon.com/ec2/ • http://aws.amazon.com/solutions/global-solution-providers/redhat/ • Cycle Scavenging • http://boinc.berkeley.edu/ • http://www.cs.wisc.edu/condor/

More Related