1 / 12

Inside Mumak

Inside Mumak. Fei Dong@duke. Architecture. Run. Cd hadoop-0.21.0/ mapred / contrib / mumak ./bin/mumak.sh Usage: mumak.sh [-- config dir] trace.json topology.json. Source code. hadoop-0.21.0/ mapred / src / contrib / mumak / src /java/org/apache/ hadoop / mapred /

mohawk
Télécharger la présentation

Inside Mumak

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Inside Mumak FeiDong@duke

  2. Architecture

  3. Run • Cd hadoop-0.21.0/mapred/contrib/mumak • ./bin/mumak.sh • Usage: mumak.sh [--config dir] trace.jsontopology.json

  4. Source code • hadoop-0.21.0/mapred/src/contrib/mumak/src/java/org/apache/hadoop/mapred/ • 21 java files, 3494 lines • Main files • 798 SimulatorTaskTracker.java • 670 SimulatorJobTracker.java • 358 SimulatorEngine.java • 354 SimulatorJobClient.java • 295 SimulatorJobInProgress.java • 137 SimulatorEventQueue.java • 110 SimulatorJobStory.java

  5. Dependency • Rumen * • org.apache.hadoop.tools.rumen.* • MapReduce • org.apache.hadoop.mapred.* • org.apache.hadoop.mapreduce.* • Others • org.apache.hadoop.conf • org.apache.hadoop.fs • org.apache.hadoop.net • org.apache.hadoop.util.Tool

  6. Main function • SimulatorEngine.java • Main loop • Init() • Set default config • start time for virtual clock • max Map/Reduce tasks per node • Setting the static mapping before removing numeric IP hosts. • create TTs based on topology.json

  7. Config Parser • JsonObjectMapperParser (behind!) • LoggedNetworkTopologytopology = new ClusterTopologyReader(new Path(topologyFile), jobConf).get(); • JobStoryProducerjobStoryProducer = new SimulatorJobStoryProducer( new Path(traceFile), cluster, firstJobStartTime, jobConf, subRandomSeed);

  8. Event • Abstract Class: SimulaterEvent LoadProbingEvent SimulaterEvent TaskAttemptCompletionEvent HeartBeatEvent JobCompleteEvent JobSubmissionEvent

  9. Event related • SimulatorEvent • SimulatorEventListener • SimulatorTaskTracker • SimulatorJobClient • SimulatorEventQueue • priority queue of events, sort by time order

  10. SimulatorJobSubmissionPolicy /** * replay the trace by following the job inter-arrival rate faithfully. */ REPLAY, /** * ignore submission time, keep submitting jobs until the cluster is saturated. */ STRESS, /** * submitting jobs sequentially. */ SERIAL; JOB_SUBMISSION_POLICY = "mumak.job-submission.policy";

  11. Future Work • Read the details of Simulator*, Rumen • Review the JIRA, discuss in the maillist • Consider where to cut in, and design • Experiment

  12. Thanks

More Related