Lecture 7: Interconnection Network

1. Lecture 7: Interconnection Network Part I: Basic DefinitionsPart II: Message Passing Multicomputers

2. Part I: Basic Definitions A network is characterized by its topology, routing algorithm, switching strategy, and flow control mechanism.

3. Basic Definitions • Topology: the physical interconnection structure of the network graph; regular (most parallel machines)or irregular (WAN). • Routing algorithm: determines which routes a message may follow through the network graph. • Switch strategy: determines how the data in a message traverses its route (circuit or packet switch) • Flow control: determines when the message, or portion of it, move along its route.

4. N/2 Butterfly ° ° ° N/2 Butterfly ° ° ° Various Interconnect Topologies

5. Routing Messages • Shared Media • Broadcast to everyone • Switched media : Needs real routing. • Circuit switching • Packet switching

6. System Architecture System Bus processor Memory Bus Adapter SCSI controller I/O Bus Network Interface Card Transmission Media Network Switch

7. Shared-Media Networks • Allow single message transmission at a time • Bus-based networks: Ethernet, Fast Ethernet • Ring-based networks: IBM Token ring, FDDI

8. Advantage simple design less expensive simple routing nature for broadcast and multicast scalable, but limited, within each segment Disadvantages fixed channel bandwidth need router or gateway to go beyond each segment limited distance span Shared-Media Network

9. Switched Network • Allow simultaneous transmission of many messages • Typical switch size: 8 to 64 ports

10. Types of Switches • Cell-based switching • fixed-size packets • e.g., ATM switches (53-byte cells) • Frame-based (Packet-based) switching • variable-size packets • e.g., Switched Ethernet (e.g., HP EtherTwist) • e.g., FDDI switch (e.g., DEC GIGAswitch) • e.g., Myrinet switch (wormhole routing)

11. Generic Switch Architecture Output buffer 4-port switch Input buffer Logical Crossbar Organization Control Unit

12. Buffer Architecture • Input buffer: • natural design: FIFO • random access buffer (more expensive) • Output buffer: more complicated • need better performance • solve output contention

13. Buffer Architecture • Dedicated buffer (for each port) • ease of routing • guaranteed service per channel • Shared buffer • better buffer utilization • one channel burst may take all buffer

14. Logical Crossbar (ATM 16-port) 155 Mbps 2.5 Gbps Bus 622 Mbps Time Division Bus

15. Other Switch Design • Shared-memory Switch • CNET Prelude switch • 2-D mesh crossbar • Myrinet, DEC GIGAswitch • Clos network (scalable) • multistage networks • Bene Network: • Washington Univ. Gigabit ATM Network

16. Cautions on Speeds • The actual application-level data rate is less than advertised speed • ATM: 155 Mbps ==> at most 134 Mbps (14%) • Switch delay: from input port to output port • Myrinet: 100 nsec (8-port) • ATM: 10-30 microseconds • WU Gigabit ATM: 10-20 microseconds

17. Packet Routing in Switched Network

18. Circuit switching • Set up; communication; release • Circuits reserved for communication • Advantages: short delays (after set up) • Disadvantage: not efficient for bursty traffic due to the long setup time.

19. Packet Switching (Datagram) • Put addresses in packets; route one by one • Switch determines the path • Deterministic: always follow same path • Adaptive: pick different paths to avoid congestion, failures • Randomized routing: pick between several good paths to balance network load • Adv: efficient; robust against failure • Disadv: delay variations; misordering possible.

20. Deterministic • Circuit established from source to destination, message picks the circuit to follow • Determined based on source and destination address • All packets follow the same route. • Adv: efficient; ordered; smaller jitter • Disadv: setup time; not robust; scalability.

21. 110 010 111 011 100 000 101 001 Deterministic Routing Examples • Mesh: • dimension-order routing • (x1, y1) -> (x2, y2) • first x = x2 -x1, • then y = y2 -y1, • Hypercube: • edge-cube routing • X = xox1x2 . . .xn -> Y = yoy1y2 . . .yn • R = X xor Y • Traverse dimensions of differing address in order • Tree: common ancestor

22. Store and Forward vs. Cut-Through • Store-and-forward: • each switch waits for the full packet to arrive in switch before sending to the next switch (good for WAN) • Cut-through: • switch examines the header, decides where to send the message, and then starts forwarding it immediately • Two approaches: (1) Virtual cut-through (2) Wormhole routing flit flit flit H D D D CRC Packet

23. H H H H H H H H a a a a a a a a b b b b b b b b c c c c c c c c Store and Forward vs. Cut-Through Node 4 (destination) Node 3 Node 2 (a) Store-and-Forward Node 1 (source) H: header a, b, c : data elements Node 4 (destination) Node 3 Node 2 (b) Cut-through (wormhole) Node 1 (source)

24. Store-and-forward buffer each packet buffer management support link-level ack good for networks with high error rate (e.g., WANs) Cut-through small buffer low latency no link-level ack good for networks with very low error rate (e.g., LANs) Switching Mechanisms

25. (1) Virtual Cut-through • To spool the blocked incoming packet into input buffer • The behavior under contention degrades to that of store-and-forward. • Requires a buffer large enough to hold the largest packet.

26. (2) Worm-hole Routing • The packet s subdivided into smaller flits. • The header flit knows where the train (packet) is going. All the data flits follow the header flit. • Different packets can be interleaved but flits from different packets cannot be mixed up. • When head of message is blocked, it leaves the tail of the message in-place along the route. • Potentially blocking other messages • Needs only buffer the piece of the packet that is sent between switches.

27. Performance Comparison • Let • L= packet length • W= channel bandwidth • D= distance (no. of nodes -1) • T store&forward= (L/W)(D+1) • Twormhole = (L/W) + (F/W)xD • If L>>F; Twormhole = (L/W) • Implication: distance insensitive

28. Store and Forward vs. Cut-Through • Advantage • Latency reduces from function of:number of intermediate switches X by the size of the packet to time for 1st part of the packet to negotiate the switches + the packet size ÷ interconnect BW

29. Congestion Control • Packet switched networks do not reserve bandwidth; this leads to contention • Solutions: • Packet discarding: If packet arrives at switch and no room in buffer, packet is discarded (e.g., UDP) • Flow control: prevent packets from entering until contention is reduced (e.g., freeway on-ramp metering lights)

30. Flow control: • Between pairs of receivers and senders; use feedback to tell sender when allowed to send next packet • Back-pressure: separate wires to tell to stop • Window: give original sender right to send N packets before getting permission to send more.(TCP) • Choke packets: Each packet received by busy switch in warning state sent back to the source via choke packet. Source reduces traffic to that destination by a fixed % (e.g., ATM)