80 likes | 84 Vues
Developing a MapReduce Application – packet dissection. How MapReduce Works?. MapReduce application is based on Hadoop Distributed FileSystem, HDFS. Steps of the MapReduce process: The client submits the job. JobTracker coordinates the job and splits into tasks.
E N D
How MapReduce Works? • MapReduce application is based on Hadoop Distributed FileSystem, HDFS. • Steps of the MapReduce process: • The client submits the job. • JobTracker coordinates the job and splits into tasks. • TaskTrackers run the tasks, here is the main map and reduce phases.
Shuffle and Sort • These two facilities are the heart of MapReduce which make Cloud Computing powerful. • Sort phase: guarantees the input to every reduce is sorted by key. • Shuffle phase: transfers the map output to the reducers as input.
A MapReduce Application – packet dissection • With Jpcap library, captures packets and writes to HDFS directly.
A MapReduce Application – packet dissection • Setup a job configuration and submit the job.
A MapReduce Application – packet dissection • The mapper filters packets with the port 1863, which is the MSN protocol.
A MapReduce Application – packet dissection • The reducer dissect the packet, and write message to output collector.
A MapReduce Application – packet dissection • See the result: