Adaptive File Transfers for Diverse Environments

1. Adaptive File Transfers for Diverse Environments Himabindu Pucha, Purdue University David G. Andersen,Carnegie Mellon University Michael Kaminsky,Intel Research Pittsburgh Michael Kozuch,Intel Research Pittsburgh

2. Goal Correctly and efficiently transfer files in wide range of scenarios

3. Network peers Goal Correctly and efficiently transfer files in wide range of scenarios • Data backup, code update • Software synchronization • Different network speeds Scenario: Data backup, code update Scenario: Software synchronization file in-place Gigabit LAN –DSL links Network Search for similar files Receiver Sender Different disk loads

4. Problem: Existing Tools Scenario-specific

5. Challenges • Resources have widely varying performance • Resource performance changes dynamically • Support receivers with different initial state • Do not require resources to be set up in advance

6. dsync: Design dsync uses all available resources effectively dsync scheduler Network Disk

7. dsync: Design • Discovers available resources using exposed backpressure information • From disk: “I’m busy writing, don’t read from me.” • From network: “I have lots of incoming packets, don’t spend time doing IO or computation.” • Schedules intelligently across available resources • Disk: use a pre-computed index and/or search entire disk using heuristics • Network:Schedule remaining chunks, least likely to be found on disk

8. dsync: Preliminary Results Throughput for 1 GB file on a 1 Gbps link dsync defers disk operations when network is faster than disk Bonus: dsync provides best of BitTorrent, rsync, scp …

9. dsync: Preliminary Results Average download time across 45 receivers, 50% similar file in-place dsync speedup: 5x vs. rsync 2x vs. SET dsync rapidly locates similar files and effectively combines them with peering dsync correctly uses backpressure to defer disk operations when network is faster than disk

10. BACKUP

11. dsync: Preliminary Results dsync correctly uses backpressure to defer disk operations when network is faster than disk

12. Network peers Goal Correctly and efficiently transfer files in wide range of scenarios • Data backup • Code update • Software synchronization • Different network speeds Scenario: Software synchronization Scenario: Data backup Scenario: Code update file in-place Gigabit LAN –DSL links Network Search for similar files Receiver Sender Different disk loads

13. Problem: Existing Tools Scenario-specific

14. dsync: Design • Discovers available resources using exposed backpressure information • From disk: “I’m busy writing, don’t read from me.” • From network: “I have lots of incoming packets, don’t spend time doing IO or computation.” • Schedules intelligently across available resources • Disk: use a pre-computed index and/or search entire disk using heuristics • Network:Schedule remaining chunks, least likely to be found on disk

15. Challenges • Correctly use resources with widely varying performance characteristics • Dynamically adapt to changes in resource performance • Support receivers with different initial state • Do not require resources to be set up in advance