Optimizing Cloud MapReduce for Processing Stream Data using Pipelining

1. Optimizing Cloud MapReduce for Processing Stream Data using Pipelining 作者:RutvikKarve，DevendraDahiphale，AmitChhajer 報告:饒展榕

2. OUTLINE • INTRODUCTION • LITERATURE SURVEY • OUR PROPOSED ARCHITECTURE • ADVANTAGES, FEATURES AND APPLICATIONS • CONCLUSIONS AND FUTURE WORK

3. INTRODUCTION • Cloud MapReduce (CMR) is a framework forprocessing large data sets of batch data in cloud. • The Mapand Reduce phases run sequentially, one after another. • Thisleads to: 1.Compulsory batch processing 2.No parallelizationof the map and reduce phases 3.Increased delays.

4. We propose a novel architecture to supportstreaming data as input using pipelining between the Mapand Reduce phases in CMR, ensuring that the output of theMap phase is made available to the Reduce phase as soon asit is produced.

5. This ‘Pipelined MapReduce’ approach leadsto increased parallelism between the Map and Reducephases; thereby 1.Supporting streaming data as input 2.Reducing delays 3. Enabling the user to take ‘snapshots’ ofthe approximate output generated in a stipulated time frame. 4. Supporting cascaded MapReduce jobs.

6. LITERATURE SURVEY • MapReduce • The model consists of two phases: a Map phaseand a Reduce phase. • Initially, the data is divided intosmaller 'splits'. • These splits of data are independent ofeach other and can hence be processed in parallel fashion. • Each split consists of a set of (key, value) pairs that formrecords.

7. B. Hadoop • Hadoop is an implementation of the MapReduceprogramming model developed by Apache. • The Hadoopframework is used for batch processing of large data setson a physical cluster of machines.

8. It incorporates adistributed file system called Hadoop Distributed FileSystem (HDFS), a Common set of commands, scheduler,and the MapReduce evaluation framework. • Hadoop is popularfor processing huge data sets, especially in socialnetworking, targeted advertisements, internet logprocessing etc.

9. C. Cloud MapReduce • Cloud MapReduce [2] is a light-weight implementation ofMapReduceprogramming model on top of the Amazoncloud OS, using Amazon EC2 instances.

10. The architectureof CMR, as described in [2] consists of one input queue,multiple reduce queues which act as staging areas forholding the intermediate (key, value) pairs produced bythe Mappers, a master reduce queue that holds thepointers to the reduce queues, and an output queue thatholds the final results.

11. D. Online MapReduce (Hadoop Online Prototype) • HOP is a modification to traditional Hadoopframework that incorporates pipelining between the Mapand Reduce phases, thereby supporting parallelismbetween these phases, and providing support forprocessing streaming data.

12. A downstream dataflow element can begin processingbefore an upstream producer finishes. • It carries out onlineaggregation of data to produce incrementally correctoutput.

13. OUR PROPOSED ARCHITECTURE • Our proposal aims at bridging this gap between heavyweightHOP and the light-weight, scalable CloudMapReduceimplementation, by providing support forprocessing stream data in Cloud MapReduce.

14. The challenges involved in theimplementation include: • 1. Providing support for streaming data at input • 2. A novel design for output aggregation • 3. Handling Reducer failures • 4. Handling windows based on timestamps.

16. The first design option: • As in CMR, the Mapper pushes each intermediate (key,value) pair to one of the reduce queues based on hashvalue of the intermediate key. • The hash function can beuser-defined or a default function provided, as in CMR.

17. 2) The second design option: • Alternatively, we can have a single queue between theMappersand the Reducers, with all the intermediate (key,value) pairs generated by all the Mappers pushed to thisintermediate queue.

18. Reducers poll this IntermediateQueuefor records. • Aggregation is carried out as follows: • Thereare a fixed number of Output splits. • Whenever a Reducerreads a (key, value) record from the IntermediateQueue, itapplies a user-defined Reduce function to the record, toproduce an output (key, value) pair.

19. It then selects anoutput split by hashing on the output key produced.

20. 3) Hybrid Approach • Have multiple ReduceQueues, each linked statically to aparticular Reducer, but instead of linking the output splitsto the Reducer statically, use hashing on the output key ofthe (key, value) pair generated by the Reducer to select anoutput split.

21. This will require fewer changes to the existingCMR architecture, but will involve static linking ofReducers to ReduceQueues.

22. ADVANTAGES, FEATURES AND APPLICATIONS • Advantages 1. Either design allows Reducers to start processing assoon as data is made available by the Mappers. Thisallows parallelism between the Map and Reducephases. 2. A downstream processing element can start processingas soon as some data is available from an upstreamelement. 3. The network is better utilized as data is continuouslypushed from one phase to the next. 4. The final output is computed incrementally. 5. Introduction of a pipeline between the Reduce phase ofone job and the Map phase of the next job will supportCascaded MapReduce jobs.

23. B. Features • Time windows 2) Snapshots 3) Cascaded MapReduceJobs

24. C. Applications • With these features, the design is particularly suited tostream processing of data. • Typically analysis and processing of webfeeds, click-streams, micro-blogging, and stock marketquotes are some of the popular and typical streamprocessing applications

25. CONCLUSIONS AND FUTURE WORK • It is alsoinherently scalable as it is cloud-based. • This also gives it a„light-weight‟ nature, as the handling of distributedresources is done by the Cloud OS. • Future work will include supporting rolling windows for obtaining outputs of arbitrary time-intervals of the input stream.