Download
three topics in parallel communications n.
Skip this Video
Loading SlideShow in 5 Seconds..
Three Topics in Parallel Communications PowerPoint Presentation
Download Presentation
Three Topics in Parallel Communications

Three Topics in Parallel Communications

143 Vues Download Presentation
Télécharger la présentation

Three Topics in Parallel Communications

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

  1. Three Topics in Parallel Communications Thesis presentation by Emin Gabrielyan Emin Gabrielyan, Three Topics in Parallel Communications

  2. Parallel communications: bandwidth enhancement or fault-tolerance? • We do not know if parallel communications were first used for fault-tolerance or for bandwidth enhancement • In 1964 Paul Baran proposed parallel communications for fault-tolerance (inspiring the design of ARPANT and Internet) • 1981 IBM invented the 8-bit parallel port for faster communication Emin Gabrielyan, Three Topics in Parallel Communications

  3. Bandwidth enhancement by parallelizing the sources and sinks • Bandwidth enhancement can be achieved by adding parallel paths • But a greater capacity enhancement is achieved if we can replace the senders and destinations with parallel sources and sinks • This is possible in parallel I/O (first topic of the thesis) Emin Gabrielyan, Three Topics in Parallel Communications

  4. Parallel transmissions in coarse-grained networks cause congestions • In coarse-grained circuit-switched HPC networks uncoordinated parallel transmissions cause congestions • The overall throughput degrades due to access conflicts on shared resources • Coordination of parallel transmissions is covered by the second topic of my thesis (liquid scheduling) Emin Gabrielyan, Three Topics in Parallel Communications

  5. Classical backup parallel circuits for fault-tolerance • Typically the redundant resource remains idle • As soon as there is a failure with the primary resource • The backup resource replaces the primary one Emin Gabrielyan, Three Topics in Parallel Communications

  6. Parallelism in living organisms • Parallelism is observed in almost every living organisms • Duplication of organs primarily serves for fault-tolerance • And as a secondary purpose, for capacity enhancement Emin Gabrielyan, Three Topics in Parallel Communications

  7. Simultaneous parallelism for fault-tolerance in fine-grained networks • A challenging bio-inspired solution is to use simultaneously all available paths for achieving fault-tolerance • This topic is addressed in the last part of my presentation (capillary routing) Emin Gabrielyan, Three Topics in Parallel Communications

  8. Fine Granularity Parallel I/O for Cluster Computers SFIO, a Striped File parallel I/O Emin Gabrielyan, Three Topics in Parallel Communications

  9. Why is parallel I/O required • Single I/O gateway for cluster computer saturates • Does not scale with the size of the cluster Emin Gabrielyan, Three Topics in Parallel Communications

  10. What is Parallel I/O for Cluster Computers • Some or all of the cluster computers can be used for parallel I/O Emin Gabrielyan, Three Topics in Parallel Communications

  11. Objectives of parallel I/O • Resistance to concurrent access • Scalability as the number of I/O nodes increases • High level of parallelism and load balance for all application patterns and all types of I/O requests Emin Gabrielyan, Three Topics in Parallel Communications

  12. Parallel I/O Subsystem Concurrent Access by Multiple Compute Nodes • No concurrent access overheads • No performsne degradation • When the number of compute nodes increases Emin Gabrielyan, Three Topics in Parallel Communications

  13. Scalable throughput of the parallel I/O subsystem • The overall parallel I/O throughput should increase linearly as the number of I/O nodes increases Throughput Number of I/O Nodes Parallel I/O Subsystem Emin Gabrielyan, Three Topics in Parallel Communications

  14. Concurrency and Scalability = Scalable All-to-All Communication Compute Nodes • Concurrency and Scalability (as the number of I/O nodes increases) can be represented by scalable overall throughput when the number of compute and I/O nodes increases All-to-All Throughput Number of I/O and Compute Nodes I/O Nodes Emin Gabrielyan, Three Topics in Parallel Communications

  15. High level of parallelism and load balance • Balanced distribution across parallel disks must be ensured: • For all types of application patterns: • Using small or large I/O requests • Continuous or fragmented I/O request patterns Emin Gabrielyan, Three Topics in Parallel Communications

  16. How parallelism is achieved? • Split the logical file into stripes • Distribute the stripes cyclically across the subfiles Logical file file2 file3 Subfiles file1 file4 file6 file5 Emin Gabrielyan, Three Topics in Parallel Communications

  17. The POSIX-like Interface of Striped File I/O #include <mpi.h> #include "/usr/local/sfio/mio.h" int _main(int argc, char *argv[]) { MFILE *f; int r=rank(); //Collective open operation f=mopen("p1/tmp/a.dat;p2/tmp/a.dat;", 5); //each process writes 8 to 14 characters at its own position if(rank==0) mwritec(f,0,"Good*morning!",13); if(rank==1) mwritec(f,13,"Bonjour!",8); if(rank==2) mwritec(f,21,"Buona*mattina!",14); mclose(f); //Collective close operation } • Using SFIO from MPI • Simple Posix like interface Emin Gabrielyan, Three Topics in Parallel Communications

  18. 0 21 13 G G o o o o d d * * m m o o r r n n i i n n g g ! ! B B o o n n j j o o u u r r ! ! B B u u o o n n a a * * m m a a t t t t i i n n a a ! ! Distribution of the global file data across the subfiles • Example with three compute nodes and two I/O nodes First subfile Global file Second subfile Emin Gabrielyan, Three Topics in Parallel Communications

  19. Impact of the stripe unit size on the load balance I/O Request • When the stripe unit size is large there is no guarantee that an I/O request will be well parallelized Logical file subfiles Emin Gabrielyan, Three Topics in Parallel Communications

  20. Fine granularity striping with good load balance I/O Request • Low granularity ensures good load balance and high level of parallelism • But results in high network communication and disk access cost Logical file subfiles Emin Gabrielyan, Three Topics in Parallel Communications

  21. Fine granularity striping is to be maintained • Most of the HPC parallel I/O solutions are optimized only for large I/O blocks (order of Megabytes) • But we focus on maintaining fine granularity • The problem of the network communication and disk access are addressed by dedicated optimizations Emin Gabrielyan, Three Topics in Parallel Communications

  22. Overview of the implemented optimizations • Disk access requests aggregation (sorting, cleaning-overlaps and merging) • Network communication aggregation • Zero-copy streaming between network and fragmented memory patterns (MPI derived datatypes) • Support of the multi-block interface efficiently optimizes application related file and memory fragmentations (MPI-I/O) • Overlapping of network communication with disk access in time (at the moment write operation only) Emin Gabrielyan, Three Topics in Parallel Communications

  23. Disk access optimizations • Sorting • Cleaning the overlaps • Merging • Input: striped user I/O requests • Output: optimized set of I/O requests • No data copy Multi-block I/O request block 1 bk. 2 block 3 6 I/O access requests are merged into 2 access1 access2 Local subfile Emin Gabrielyan, Three Topics in Parallel Communications

  24. Network Communication Aggregation without Copying From: application memory • Striping across 2 subfiles • Derived datatypes on the fly • Contiguous streaming Logical file To: remote I/O nodes Remote I/O node 1 Remote I/O node 2 Emin Gabrielyan, Three Topics in Parallel Communications

  25. MPI MPI MPI MPI Functional Architecture mread mreadc mreadb mwritec mwriteb mwrite mrw (cyclic distribution) sfp_rflush sfp_wflush • Blue: Interface functions • Green: Striping functionality • Red: I/O request optimizations • Orange: Network communication and relevant optimizations • bkmerge: overlapping and aggregation • mkbset: creates on the fly MPI derived datatypes sfp_readc sfp_writec SFIO library on compute node sfp_rdwrc (request caching) sfp_read sortcache sfp_write flushcache bkmerge sfp_readb sfp_writeb sfp_waitall mkbset SFP_CMD _WRITE I/O Node I/O Listener SFP_CMD_ BREAD SFP_CMD_ BWRITE SFP_CMD _READ Emin Gabrielyan, Three Topics in Parallel Communications

  26. Optimized throughput as a function of the stripe unit size • 3 I/O nodes • 1 compute node • Global file size: 660 Mbytes • TNET • About 10 MB/s per disk Emin Gabrielyan, Three Topics in Parallel Communications

  27. All-to-all stress test on Swiss-Tx cluster supercomputer • Stress test is carried out on Swiss-Tx machine • 8 full crossbar 12-port TNet switches • 64 processors • Link throughput is about 86 MB/s Emin Gabrielyan, Three Topics in Parallel Communications

  28. SFIO on the Swiss-Tx cluster supercomputer • MPI-FCI • Global file size: up to 32 GB • Mean of 53 measurements for each number of nodes • Nearly linear scaling with 200 bytes stripe unit ! • Network is a bottleneck above 12 nodes Emin Gabrielyan, Three Topics in Parallel Communications

  29. Liquid scheduling for low-latency circuit-switched networks Reaching liquid throughput in HPC wormhole switching and in Optical lightpath routing networks Emin Gabrielyan, Three Topics in Parallel Communications

  30. Upper limit of the network capacity • Given is a set of parallel transmissions • and a routing scheme • The upper limit of network’s aggregate capacity is its liquid throughput Emin Gabrielyan, Three Topics in Parallel Communications

  31. Distinction: Packet Switching versus Circuit Switching • Packet switching is replacing circuit switching since 1970 (more flexible, manageable, scalable) • New circuit switching networks are emerging (HPC clusters, Optical switching) • In HPC wormhole routing targets extremely low latency requirements • In optical network packet switching is not possible due to lack of technology Emin Gabrielyan, Three Topics in Parallel Communications

  32. Message Sink Message Source Coarse-Grained Networks • In circuit switching the large messages are transmitted entirely (coarse-grained switching) • Low latency • The sink starts receiving the message as soon as the sender starts transmission Fine-Grained Packet switching Coarse-grained Circuit switching Emin Gabrielyan, Three Topics in Parallel Communications

  33. Parallel transmissions in coarse-grained networks • When the nodes transmit in parallel across a coarse-grained network in uncoordinated fashion congestion may occur • The resulting throughput can be far below the expected liquid throughput Emin Gabrielyan, Three Topics in Parallel Communications

  34. Congestions and blocked paths in wormhole routing Source3 • When the message encounters a busy outgoing port it waits • The previous portion of the path remains occupied Sink2 Source1 Source2 Sink1 Sink3 Emin Gabrielyan, Three Topics in Parallel Communications

  35. Hardware solution in Virtual Cut-Through routing Source3 • In VCT when the port is busy • The switch buffers the entire message • Much more expensive hardware than in wormhole switching Sink2 Source1 buffering Source2 Sink1 Sink3 Emin Gabrielyan, Three Topics in Parallel Communications

  36. Other hardware solutions • In optical networks OEO conversion can be used • Significant impact on the cost (vs. memory-less wormhole switch and MEMS optical switches) • Affecting the properties of the network (e.g. latency) Emin Gabrielyan, Three Topics in Parallel Communications

  37. Application level coordinated liquid scheduling • Liquid scheduling is a software solution • Implemented at the application level • No investments in network hardware • Coordination between the edge nodes is required • Network topology knowledge is assumed Emin Gabrielyan, Three Topics in Parallel Communications

  38. Example of a simple traffic pattern • 5 sending nodes (above) • 5 receiving nodes (below) • 2 switches • 12 links of equal capacity • Traffic consist of 25 transfers Emin Gabrielyan, Three Topics in Parallel Communications

  39. Round robin schedule of all-to-all traffic pattern • First, all nodes simultaneously send the message to the node in front • Then, simultaneously, to the next node • etc Emin Gabrielyan, Three Topics in Parallel Communications

  40. Throughput of round-robin schedule • 3rd and 4th phases require each two timeframes • 7 timeframes are needed in total • Link throughput = 1Gbps • Overall throughput = 25/7x1Gbps = 3.57Gbps Emin Gabrielyan, Three Topics in Parallel Communications

  41. A liquid schedule and its throughput • 6 timeframes of non-congesting transfers • Overall throughput = 25/6x1Gbps = 4.16Gbps Emin Gabrielyan, Three Topics in Parallel Communications

  42. Problem of liquid scheduling • Building liquid schedule for arbitrary traffic of transfers • Problem of partitioning of the traffic into minimal number of subsets consisting of non-congesting transfers • Timeframe = a subset of non-congesting transfers Emin Gabrielyan, Three Topics in Parallel Communications

  43. Definitions of our mathematical model • Transfer is a set of links lying on the path of the transmission • Load of a link is the number of transfers in the traffic using that link • Most loaded links are called bottlenecks • Duration of the traffic is the load of its bottlenecks Emin Gabrielyan, Three Topics in Parallel Communications

  44. Teams = non-congesting transfers using all bottleneck links • The shortest possible time to carry out the traffic is the active time of the bottleneck links • Then the schedule must keep the bottleneck links busy all the time • Therefore the timeframes of a liquid schedule must consist of transfers using all bottlenecks team bottlenecks not a team Emin Gabrielyan, Three Topics in Parallel Communications

  45. Retrieval of teams without repetitions by subdivisions • Teams can be retrieved without repetitions by recursive partitioning • By a choice of a transfer all teams are divided into teams using that transfer and teams not using it • Each halves can be similarly sub divided until individual teams are retrieved Emin Gabrielyan, Three Topics in Parallel Communications

  46. Teams use all bottlenecks: retrieving teams of traffic skeleton • Since teams must use transfers using the bottleneck links • We can first create teams using only such transfers (traffic skeleton) • Chart: fraction of the traffic skeleton Emin Gabrielyan, Three Topics in Parallel Communications

  47. Optimization by first retrieving the teams of the skeleton • Speedup: by skeleton optimization • Reducing the search space 9.5 times Emin Gabrielyan, Three Topics in Parallel Communications

  48. Liquid schedule assembling from retrieved teams • By relying on efficient retrieval of full teams (subsets of non-congesting transfers using all bottlenecks) • We assemble liquid schedule by trying together different combinations of teams • Until all transfers of the traffic are used Emin Gabrielyan, Three Topics in Parallel Communications

  49. Liquid schedule assembling optimizations (reduced traffic) • Proved. If we remove a team from a traffic, new bottlenecks can emerge • New bottlenecks add additional constraints on the teams of the reduced traffic • Proved. A liquid schedule can be assembled if we use teams of the reduced traffic (instead of constructing teams of the initial traffic from the remaining transfers) • Proved. A liquid schedule can be assembled by considering only saturated full teams Emin Gabrielyan, Three Topics in Parallel Communications

  50. Liquid schedule construction speed with our algorithm • 360 traffic patterns across Swiss-Tx network • Up to 32 nodes • Up to 1024 transfers • Comparison of our optimized construction algorithm with MILP method (optimized for discrete optimization problems) Emin Gabrielyan, Three Topics in Parallel Communications