Synchronous Computation

1. Synchronous Computation Barrier A barrier is inserted at the point in each process where it must wait. All processes can continue from this point when all the processes have reached it.

2. Synchronous Computation

3. Barrier Implementation Counter implementation arrival phase departure phase

4. Barrier Implementation for(i=0; i<n ; i++) recv(Pany); for(i=0; i<n ; i++) send(Pi); send(Pmaster); recv(Pmater);

5. Barrier Implementation Tree implementation O(n)

6. Barrier Implementation Butterfly Barrier At stage s, process i synchronizes with process i+2s-1 if n is a power of 2. O(log n)

7. Barrier Implementation Local Synchronization Process Pi-1 recv(Pi); send(Pi); Process Pi send(Pi-1); send(Pi+1); recv(Pi-1); recv(Pi+1); Process Pi+1; recv(Pi); send(Pi);

8. Barrier Implementation Deadlock Process Pi-1 sendrecv(Pi); Process Pi sendrecv(Pi-1); sendrecv(Pi+1); Process Pi+1 sendrecv(Pi);

9. Data Parallel Computation Data parallel programming is particularly convenient for two reasons. The first is its ease of programming. The second is that it can scale easily to larger problem size. for(i=0 ; i<n ; i++) a[i] = a[i] + k;

10. Data Parallel Computation i=myrank; a[i] = a[i] + k; barrier(mygroup); forall(i=0 ; i<n ; i++){ body } forall (i=0 ; i<n ; i++) a[i] = a[i] + k;

11. Prefix Sum Problem O(n2) for(i=0 ; i<n ; i++) { sum[i] = 0; for(j=0 ; j<=i ; j++) sum[i] = sum[i] + x[j]; }

12. Prefix Sum Problem O(n log n) for(j=0 ; j<log(n) ; j++) for(i=2^j ; i<n ; i++) x[i] = x[i] + x[i-2^j]; for(j=0 ; j<log(n) ; j++) forall(i=0 ; i<n ; i++) if( i>=2^j ) x[i] = x[i] + x[i-2^j];

13. Synchronous Iteration for(j=0 ; j<n ; j++) forall(i=0 ; i<N ; i++){ body(i); } for(j=0 ; j<n ; j++){ i=myrank; body(i); barrier(mygroup); }

14. Solving Linear Equations by iterations Diagonally dominant

15. Solving Linear Equations by iterations Termination

16. Solving Linear Equations by iterations Sequential Code for(i=0 ; i<n ; i++) x[i] = b[i]; for(iteration=0 ; iteration<limit ; iteration++){ for(i=0 ; i<n ; i++){ sum=0; for(j=0 ; j<n ; j++) if( i!= j) sum = sum + a[i][j] * x[j]; new_x[i] = (b[i] - sum) / a[i][i]; } for(i=0 ; i<n ; i++) x[i] = new_x[i]; }

17. Solving Linear Equations by iterations for(i=0 ; i<n ; i++) x[i] = b[i]; for(iteration=0 ; iteration<limit ; iteration++){ for(i=0 ; i<n ; i++){ sum=-a[i][i]*x[i]; for(j=0 ; j<n ; j++) sum = sum + a[i][j] * x[j]; new_x[i] = (b[i] - sum) / a[i][i]; } for(i=0 ; i<n ; i++) x[i] = new_x[i]; }

18. Solving Linear Equations by iterations Parallel Code x[i] = b[i]; for(iteration=0 ; iteration<limit ; iteration++){ sum = -a[i][i]*x[i]; for(j=1 ; j<n ; j++) sum = sum + a[i][j]*x[j]; new_x[i] = (b[i]-sum) / a[i][i]; broadcast_receive(&new_x[i]); global_barrier(); }

19. Solving Linear Equations by iterations x[i] = b[i]; iteration = 0; do{ iteration++; sum = -a[i][i]*x[i]; for(j=1 ; j<n ; j++) sum = sum + a[i][j]*x[j]; new_x[i] = (b[i]-sum) / a[i][i]; broadcast_receive(&new_x[i]); }while(tolerance() && (iteration<limit));

20. Solving Linear Equations by iterations

21. Solving Linear Equations by iterations Analysis computation communication

22. Solving Linear Equations by iterations tstartup=10,000 tdata=50

23. Heat Distribution Problem

24. Heat Distribution Problem for(iteration=0 ; iteration<limit ; iteration++){ for(i=1; i<n ; i++) for(j=1; j<n ; j++) g[i][j] = 0.25 *(h[i-1][j]+h[i+1][j]+h[i][j-1]+h[i][j+1]); for(i=0 ; i<n ; i++) for(j=1 ; j<n ; j++) h[i][j] = g[i][j]; }

25. Heat Distribution Problem do{ for(i=1; i<n ; i++) for(j=1; j<n ; j++) g[i][j] = 0.25 *(h[i-1][j]+h[i+1][j]+h[i][j-1]+h[i][j+1]); for(i=0 ; i<n ; i++) for(j=1 ; j<n ; j++) h[i][j] = g[i][j]; continue = false; for(i=0 ; i<n ; i++) for(j=0 ; j<n ; j++) if(!converged(i,j)){ continue = true; break; } }while(continue == true);

26. Heat Distribution Problem for(iteration=0 ; iteration<limit ; iteration){ g = 0.25*(w+x+y+z); send(&g, Pi-1,j); send(&g, Pi+1,j); send(&g, Pi,j-1); send(&g, Pi,j+1); recv(&w, Pi-1,j); recv(&x, Pi+1,j); recv(&y, Pi,j-1); recv(&z, Pi,j+1); }

27. Heat Distribution Problem iteration=0; do{ iteration++; send(&g, Pi-1,j); send(&g, Pi+1,j); send(&g, Pi,j-1); send(&g, Pi,j+1); recv(&w, Pi-1,j); recv(&x, Pi+1,j); recv(&y, Pi,j-1); recv(&z, Pi,j+1); }while((!converged(i,j))||(iteration == limit)); send(&g, &i, &iteration, Pmaster);

28. Heat Distribution Problem

29. Heat Distribution Problem if(last_row) w=bottom_value; if(first_row) x=top_value; if(first_column)y=left_value; if(last_column)z=right_value; iteration=0; do{ iteration++; g=0.25*(w+x+y+z); if !(first_row) send(&g, Pi-1,j); if !(last_row) send(&g, Pi+1,j); if !(first_column) send(&g, Pi,j-1); if !(last_column) send(&g, Pi,j+1); if !(last_row) recv(&w, Pi-1,j); if !(first_row) recv(&x, Pi+1,j); if !(first_column) recv(&y, Pi,j-1); if !(last_column) recv(&z, Pi,j+1); }while((!converged) || (iteration == limit)); send( &g, &i, &j, iteration, Pmaster);

30. Heat Distribution Problem Partitioning

31. Heat Distribution Problem

32. Heat Distribution Problem for(i=1 ; i<m ; i++) for(j=1 ; j<n/p ; j++) g[i][j]= 0.25*(h[1][j]+h[i+1][j]+h[i][j-1]+h[i][j+1]); for(i=1 ; i<m ; i++) for(j=1 ; j<n/p ; j++) h[i][j] = g[i][j]; send(&g[1][1], &m, Pi-1); send(&g[1][m], &m, Pi+1); recv(&h[1][0], &m, Pi-1); recv(&h[1][m+1], &m, Pi+1);

33. Heat Distribution Problem

34. Heat Distribution Problem if( (myid%2) == 0){ send(&g[1][1], &m, Pi-1); recv(&h[1][0], &m, Pi-1); send(&g[1][m], &m, Pi+1); send(&h[1][m+1], &m, Pi+1); } else{ recv(&h[1][0], &m, Pi-1); send(&g[1][1], &m, Pi-1); recv(&h[1][m+1], &m, P+1); send(&g[1][m], &m, Pi+1); }