Particle Advection Defined & Usage

1. Particle Advection Defined & Usage

2. Particle advection is a foundational visualization algorithm • Advecting particles creates integral curves

3. Particle advection is a foundational visualization algorithm Courtesy C. Garth, Kaiserslautern

4. Particle advection is a foundational visualization algorithm Streamlines in the “fish tank”

5. Particle advection is a foundational visualization algorithm Poincaré analysis of a tokamak Image credit: A. Sanderson, Univ. Utah

6. Particle advection is a foundational visualization algorithm Stream surface around a tokamak

7. Lagrangian Methods • Visualize manifolds of maximal stretching in a flow, as indicated by dense particles • Finite-Time Lyapunov Exponent (FTLE) Courtesy Garth

8. Lagrangian Methods • Visualize manifolds of maximal stretching in a flow, as indicated by dense particles • Forward in time: FTLE+ indicates divergence • Backward in time: FTLE-indicates convergence www.vacet.org Courtesy Garth

9. Particle Advection In A Parallel Setting

10. Parallel Particle Advection: How ToLoad Balance? • N particles (P1, P2, … Pn), M Processing Elements/PEs (PE1, …, PEm) • (Processing Element = instance of your program) • Each particle takes a variable number of steps, S1, S2, … Sn • Total number of steps is ΣSi • We cannot do less work than this (ΣSi) • Goal: Distribute the ΣSi steps over M PEs such that problem finishes in minimal time

11. Parallel Particle Advection: How ToLoad Balance? • Goal: Distribute the ΣSi steps over M PEs such that problem finishes in minimal time • Sounds sort of like a bin-packing problem, but… • particles can move from PE to PE (i.e. work can move from bin to bin) • path of particle is data dependent and unknown a priori (i.e. we don’t know Si beforehand) • big data significantly complicates this picture…. • … data may not be readily available, introducing starvation

12. Advecting particles Decomposition of large data set into blocks on filesystem ? What is the right strategy for getting particle and data together?

13. Strategy: load blocks necessary for advection Decomposition of large data set into blocks on filesystem Go to filesystem and read block

14. Strategy: load blocks necessary for advection Decomposition of large data set into blocks on filesystem

15. “Parallelize over Particles” • Basic idea: particles are partitioned over PEs, blocks of data are loaded as needed. • Positives: • Indifferent to data size: a serial program can process data of any size • Trivial parallelization (partition particles over processors) • Negative: • Redundant I/O (both across PEs and within a PE) is a significant problem.

16. “Parallelize over data” strategy:parallelize over blocks and pass particles PE2 PE1 PE4 PE3

17. “Parallelize over Data” • Basic idea: data is partitioned over PEs, particles are communicated as needed. • Positives: • Only load data one time • Great for in situ processing • Negative: • Starvation!

18. Contracts are needed to enable these different processing techniques • Parallelize-over-seeds • One execution per block • Only limited data available at one time • Parallelize-over-data • One execution total • Entire data set available at one time

19. Both parallelization schemes have serious flaws. • Two approaches: Parallelize over particles Parallelize over data Hybrid algorithms

20. The master-slave algorithm is an example of a hybrid technique. • Uses “master-slave” model of communication • One process has unidirectional control over other processes • Algorithm adapts during runtime to avoid pitfalls of parallelize-over-data and parallelize-over-particles. • Nice property for production visualization tools. • (Implemented in VisIt) D. Pugmire, H. Childs, C. Garth, S. Ahern, G. Weber, “Scalable Computation of Streamlines on Very Large Datasets.” SC09, Portland, OR, November, 2009

21. Master P0 P1 P2 P3 Master Master Master P4 P5 P6 P7 P8 P9 P10 P11 P12 P13 P14 P15 Slave Slave Slave Slave Slave Slave Slave Slave Slave Slave Slave Slave Master-Slave Hybrid Algorithm • Divide processors into groups of N • Uniformly distribute seed points to each group • Master: • Monitor workload • Make decisions to optimize resource utilization • Slaves: • Respond to commands from Master • Report status when work complete

22. Master Process Pseudocode Master() { while ( ! done ) { if ( NewStatusFromAnySlave() ) { commands = DetermineMostEfficientCommand() for cmd in commands SendCommandToSlaves( cmd ) } } } What are the possible commands?

23. Commands that can be issued by master • Assign / Loaded Block • Assign / Unloaded Block • Handle OOB / Load • Handle OOB / Send • OOB = out of bounds Master Slave Slave is given a particle that is contained in a block that is already loaded

24. Commands that can be issued by master • Assign / Loaded Block • Assign / Unloaded Block • Handle OOB / Load • Handle OOB / Send • OOB = out of bounds Master Slave Slave is given a particle and loads the block

25. Load Commands that can be issued by master • Assign / Loaded Block • Assign / Unloaded Block • Handle OOB / Load • Handle OOB / Send • OOB = out of bounds Master Slave Slave is instructed to load a block. The particle advection in that block can then proceed.

26. Send to J Commands that can be issued by master • Assign / Loaded Block • Assign / Unloaded Block • Handle OOB / Load • Handle OOB / Send • OOB = out of bounds Master Slave Slave J Slave is instructed to send the particle to another slave that has loaded the block

27. Master Process Pseudocode Master() { while ( ! done ) { if ( NewStatusFromAnySlave() ) { commands = DetermineMostEfficientCommand() for cmd in commands SendCommandToSlaves( cmd ) } } }

28. Driving factors… • You don’t want PEs to sit idle • (The problem with parallelize-over-data) • You don’t want PEs to spend all their time doing I/O • (The problem with parallelize-over-seeds) •  you want to do “least amount” of I/O that will prevent “excessive” idleness.

29. The heuristics behind the heuristics • Rules: • If no PE has the block, then load it! • If a PE does have the block… • If it is not busy, then give it the particle • If it is busy, then wait “a while”. • If you’ve waited a “long time”, then load it yourself.

30. Master-slave in action S0 0: Read S0 S1 S3 0: Read S1 1: Pass S2 S4 1: Pass 1: Read

31. Algorithm Test Cases • Core collapse supernova simulation • Magnetic confinement fusion simulation • Hydraulic flow simulation

32. Workload distribution in parallelize-over-data Starvation

33. Workload distribution in parallelize-over-particles Too much I/O

34. Workload distribution in master-slave algorithm Just right

35. Workload distribution in supernova simulation Parallelization by: Data Particles Hybrid Colored by PE doing integration

36. Astrophysics Test Case: Total time to compute 20,000 Streamlines Uniform Seeding Non-uniform Seeding Seconds Seconds Number of procs Number of procs Particles Hybrid Data

37. Astrophysics Test Case: Number of blocks loaded Uniform Seeding Non-uniform Seeding Blocks loaded Blocks loaded Number of procs Number of procs Data Hybrid Particles

38. Summary: Master-Slave Algorithm • First ever attempt at a hybrid algorithm for particle advection • Algorithm adapts during runtime to avoid pitfalls of parallelize-over-data and parallelize-over-particles. • Nice property for production visualization tools. • Implemented inside VisIt visualization and analysis package.

39. Benefits of an Extended Memory Hierarchy

40. Emerging I/O Configuration Common I/O Configuration Emerging I/O Configuration

41. Simulation Codes SimulationCode Checkpoint/Restart Node GlobalFile System SimulationCode Checkpoint/Restart SSD Node

42. Parallelize-over-Seeds Algorithm IntegrationI/O Streamlines GlobalFile System Memory Cache Node

43. Parallelize-over-Seeds Algorithm with the Extended Memory Hierarchy IntegrationI/O Streamlines GlobalFile System Memory Cache Local File System Node

44. I/O Configurations

48. A Real World Particle Advection System

49. Goal • Efficient code for a variety of particle advection based techniques • Cognizant of use cases with >>10K particles. • Need handling of every particle, every evaluation to be efficient. • Want to support diverse flow techniques: flexibility/extensibility is key. • Fit within data flow network design (i.e. a filter)

50. Motivating examples of system • FTLE • Stream surfaces • Streamline • Dynamical Systems (e.g. Poincaré Maps) • Statistics based analysis • + more