Parallelism is Everywhere – So How Do We Make It Accessible?

1. Parallelism is Everywhere –So How Do We Make It Accessible? Daniel Ernst Department of Computer Science The University of Wisconsin – Eau Claire

2. A Little Background … • “Grew Up” (Graduate School) in Computer Architecture • Efficient, fast, low-power processors (2 ISCA, 1 MICRO papers, +) • “Dabbled” in supercomputing • (Becoming a bigger part of my job now…) • Became a CS educator • Now • Trying to make sense of how these are all coming together

3. Moore’s Dividend

4. Where Did All The Performance Go?Software Development, c. 1950-2005 “What Andy giveth, Bill taketh away.” Figure by Jim Larus [CACM 09]

5. Can Multi-Core Supplant Moore’s “Dividend”? (for individual programs) • Double number of cores instead of speed • NO, at least not without major innovation • Sequential code base will not scale without lots of help • Lack of parallel algorithms (and/or poor knowledge of them) • Difficult (or at least different) programming challenges • Fewer abstractions currently available

6. Parallel Algorithms “In the context of sequential algorithms, it is standard practice to design more complex algorithms that outperform simpler ones (for example, by implementing a balanced tree instead of a list). For non-blocking algorithms, however, implementing more complex data structures has been prohibitively difficult.” [Herlihy, Luchangco, Moir, Scherer, PODC 2003] (Discussing a concurrent red-black tree)

7. Sequential to Parallel – A History • Existing code is sequential • We tried parallel compilers in the ‘80s-’90s • Can get good code for a few specific cases, but not generally • We tried to get the hardware to extract parallelism in 90’s-00’s • Dynamic mechanisms cannot find more than 2–4x parallelism • The issue is with how programmers fundamentallythink about problems • Not language specific • If we’re training programmers to think only in sequence, how well will they be prepared for dealing with explicit parallelism? Failed Failed

8. Fundamental Take-aways • Explicit parallelism is important, and is getting more so • Being able to identify it will be an increasingly marketable skill • Understanding the underlying architectures is also becoming more important • Necessary to have at least basic understanding to extract reasonable amounts of parallelism • Hybrid machines will make this critical if you want good performance • ESPECIALLY before the tools catch up (but still after)

9. Programming Futures • There is a lack of consensus on the parallel programming model • Data parallelism (Vector) • Unstructured (thread) parallelism • (Edward Lee – “The Problem with Threads”) • Speculative threading? • Message passing • Lots of ongoing research with not much resolved • Moral:Everyone in the CS field needs to be prepared to learnabout new parallel programming methods and tools.

10. State of the Game Educators This Talk Industry This (slightly modified) graph shamelessly stolen from Jim Larus

11. An Opportunistic Approach • Typically,he/she who develops functionality to use fully use the hardware capabilities “wins” • Google, facebook, intel, Microsoft, etc. all took advantage of technological inflection points • So, if we don’t know what to teach yet, how do we prepare our students to be leaders?

12. Some Context • UW – Eau Claire “Founded in 1916, UW-Eau Claire is one of the Midwest’s top public universities. UW-Eau Claire provides students with small classes led by talented professors, cutting-edge programs, hands-on research and life-enhancing study abroad opportunities. Motto:Excellence. Our measure, our motto, our goal. • What that actually means: • Approx. 11k students, only 500 are grad students • Focus is undergraduate engagement • Teaching excellence • Undergraduate research

13. UW – Eau Claire Computer Science • ABET Accredited comprehensive CS major • 6 faculty (+1 vacancy), 1 part-time academic staff • Approximately 40-50 graduates per year in 3 majors: • Comprehensive CS • Software Engineering • Computer Engineering (CS degree with CE emphasis) • We’re filthy rich (sort of) • Constituencies / Industrial Partners / Landing Spots • Healthcare, insurance (Marshfield Clinic, NW Mutual) • Some systems development (Cray/IBM)

14. UW – Eau Claire Computer Science • Focus is on keeping an innovative curriculum • A majority of our scholarship is within this scope • SIGCSE-related activity is large • total of 31 papers and 7 panels presented in SIGCSE sponsored conferences and journals since 2002 • The department has recently been nominated three times as the University of Wisconsin Teaching Department of the Year (2001, 2002, 2003) • Curricular Basics: • Java early, C/C++ later, plus a full PL course • Full P&H Comp Org. • Late: strong software engineering coursework along with robust systems classes (OS/Networks) Not for the casual student

15. Parallelism in the Curriculum – Current • Many (most?) schools treat parallel computing as an independent topic, addressed in the senior year as an elective.

16. Curricular Goals at UW-EC Students should gain: • a generic understanding of computational concurrency and parallelism. • see parallelism in problems • be able to exploit it in a reasonable way • exposure to parallelism as an orthogonal method with respect to other standard curricular items. • Don’t just teach parallelism by itself – integrate it in with other elements as simply another tool in the programming box. • Teaching it in context can lower time overhead, as well • confidence in using parallelism as a tool.

17. Principles for Implementation • Simplicity • Keep to easy (deterministic) and natural concepts • Ubiquity • More than a “1-shot” issue • Accessibility • Don’t make the learning curve steeper than it has to be

18. Simplicity Keeping Parallelism Simple • First, tweak the underlying concept of a computer • A “collection of brains” • The concepts behind basic parallelism are simple • Sharing work • Different tasks • Split up data • Especially if they’re given a more natural context

19. Simplicity Natural Parallelism/Concurrency • We have very natural allies in the educational graphics/animation toolsets • Alice, Scratch, etc. • These tools make basic concurrency natural and largely deterministic • easier to debug!

20. Simplicity Another Natural Example – Swarm Intelligence • “Creative” Monte Carlo solution to TSP • Instantiate multiple agents to probabilistically discover the shortest trip • “Threads as Entities” • Each agent (“ant”) operates in its own thread • ants leave pheromone trails along arcs to influence later ants • Arc information needs to be synchronized • Orthogonal topics: • Monte Carlo • heuristics

21. Simplicity Other “Clean” Parallel Examples

22. Ubiquity Parallelism Everywhere  Parallelism the Norm • Unlike the past, sequential code written today is likely to only see very small (if any) gains from hardware upgrades. • As the number of cores increases, non-parallel performance will become relatively more unacceptable in practice. • Exploiting parallelism needs to be a first-class citizen • (or at least second-class) • not an item studied once and then not used again. • Ubiquitous

23. Ubiquity An Integrated Model at UW-EC • CS1 – Decomposition and Assignment • Introduce methods for recognizing obvious parallelism • Practice with simple, natural, and integrated examples • CS2 – Basic Orchestration • Continued practice with obvious parallelism • Introduce communication • Algorithms – Parallelism Limits • Extend algorithm analysis to parallel algorithms • Introduce more complicated communication patterns • Operating Systems – Synchronization Constructs • “The usual topics “

24. Ubiquity After a Reality Check… • CS2 – Decomposition and Assignment • Introduce methods for recognizing obvious parallelism • Practice with simple, natural, and integrated examples • CS2/Algorithms – Basic Orchestration • Continued practice with obvious parallelism • Introduce communication • Algorithms/OS – Parallelism Limits • Extend algorithm analysis to parallel algorithms • Introduce more complicated communication patterns • Operating Systems – Synchronization Constructs • “The usual topics “

25. Ubiquity CS2 Possibilities – Image Mosaic • “Nifty Assignment” (Pattis, SIGCSE 2005) • create an image out of tiles ofsmaller images • tiles are independent (unless you want it to look good!) • Basic parallelism is easy, butgood quality needs synchronization • Payoff! • Can vary synchronization difficulty • Orthogonal topics: • searching/sorting/matching

26. Ubiquity The Role of Architecture • Understanding how a program “lays out” on a machine. • What is it capable of? • What are the bottlenecks? • My take thus far: • Decreased emphasis on assembly programming • Slightly decreased emphasis on microarchitecture • (Depending on program needs) • Increased emphasis on system-level view (interaction between processor and memory)

27. Accessibility Keeping Parallelism Accessible • For each individual curriculum, the choice of language/platform has a significant effect on accessibility. • At UW-EC, we are largely a “Java shop”, with some C/C++ • To keep learning curve gentle, we try to operate within that framework • Java.util.concurrent / OpenMP • Taking the time to introduce a new platform eats far too much time and can lead to projection of learning difficulties* • (and they’re just going to change, anyways) The concepts are more important than the platform *YMMV

28. Accessibility The Delicate Balance While we want to have student learning to be as platform-agnosticas possible… …they also need to learn real skills for the workforce… …and they need to practice with the concepts somehow!

29. Current Challenges • General Workload • Especially given the ubiquity argument, there’s a lot of material that needs to be created/re-worked in new contexts • How do we motivate instructors? • Faculty Readiness • At least anecdotally, the percentage of faculty who feel comfortable teaching parallel computing is painfully small • How can we help with faculty accessibility? • Future Uncertainty • Given that we’re not quite sure what future programming models will look like – what topics should we be teaching?

30. Current Challenges (More?!?) • Availability of Tools/Language Support for Parallelism • There are very few language/toolset combinations that collectively fulfill the major criteria for curricular languages: • Language used in industry (or related to one) • Supports OO • Good support for “easy” parallelism • Not syntactic suicide

31. The Reality of Implementation • If change is really going to be cross-curriculum, it requires that there is significant buy-in from everyfaculty/staff member that teaches any course that includes the new material. • This is a serious professional development challenge

32. EAPF – Addressing The Challenges Educational Alliance for a Parallel Future • Combined effort of Industry: • Intel, nVidia, AMD, Microsoft, etc. • and Academia • Members range from research (UIUC/Berkeley) to supercomputing experts to small college educators • Goals: • Increase awareness • Encourage and support innovation in training and tools • Keep an eye out for more…

33. Thank You!Q & A