SlideServe
Télécharger Obtenir Pro
Menu
Accueil Pro
Préférences
ApparenceLight Aide et commentaires
© SlideServe · v2026

Comprehensive Guide to Grid Computing with BatchMake

elmo avatar
elmo
26 diapositives · 352 Views · Aug 15, 2014

Delve into BatchMake, a tool for grid computing, data processing, and algorithm design. Learn about its origins, workflow, command line execution, and integration with Condor. Discover how BatchMake simplifies command line arguments and job management. Explore Condor's workload management for compute-intensive tasks and scripting examples. Monitor Condor jobs and dive into reporting with BatchMake for a seamless development experience.

#comprehensive guide to grid computing with batchmake #comprehensive guide to grid computing with batchmake ppt #powerpoint #presentation
Comprehensive Guide to Grid Computing with BatchMake - slide 1
01 / 26Comprehensive Guide to Grid Computing with BatchMake
1 / 17

Télécharger la présentation

Comprehensive Guide to Grid Computing with BatchMake
Une image / un lien ci-dessous est fourni (tel quel) pour télécharger la présentation

Politique de téléchargement : le contenu du site vous est fourni TEL QUEL pour votre information et votre usage personnel ; il ne peut être vendu, concédé sous licence ni partagé sur d’autres sites sans l’accord de son auteur.Le contenu vous est fourni TEL QUEL, pour votre information et votre usage personnel uniquement. Téléchargez la présentation en cliquant sur ce lien. Lors du téléchargement, si vous ne parvenez pas à télécharger une présentation, l’éditeur a peut-être supprimé le fichier de son serveur.Pendant le téléchargement, si vous ne parvenez pas à obtenir une présentation, le fichier a peut-être été supprimé par l’éditeur.

F I N

Plus de contenu associé

Transcription de la présentation

  1. Grid Computing with BatchMake Julien JomierKitware Inc.

  2. Motivation Data Processing Reports & Analysis Algorithm Design & Implementation

  3. Outline • Introduction to BatchMake • Grid Computing with BatchMake • Reporting with the BatchBoards • BatchMake with Slicer • Future plans

  4. BatchMake Origins • Started in 2004 at UNC (CADDLab) • A2D2 from NLM • Kitware funds: KWGrid • BW-NAC: Batchboards • Na-Mic • STTR Kitware-UNC (Styner)

  5. Introduction to BatchMake • Batch script generator • Process of large number of datasets • Common Language (CMake-like) • Generation for: - Shell script - Condor - KWGrid • Run locally or on grid • Cross platform command line and GUI tool • C++ with ITK and FTLK

  6. BatchMake Workflow CommandLine Executable Local Execution ApplicationWrapper Shell Script BatchMake KWGrid Script Script Condor Script

  7. Application Wrapper • Describe command line arguments- Manually (GUI)- Automatically using itkMetaCommand • Currently using internal XML format • .bmm files (BatchMake Modules) • Switching to Slicer’s ModuleDescriptionParser- importing Slicer Module automatically

  8. MetaCommand • C++ class for command line parsing • Output recognized by - BatchMake (-vxml) - Slicer (--xml) • command.SetOption("compileScript","c",false,"Compile a script"); command.AddOptionField("compileScript","filename",MetaCommand::STRING,true); • http://www.itk.org/Wiki/MetaIO • Insight/Utilities/MetaIO

  9. Example Script 1 -SetApp(imSeg @ImageSegmentation) 2 -SetAppOption(imSeg.threshold 10) 3 - 4 -Set(workdir '/BatchMake/Demo/Ellipse_Images') 5 -ListFileInDir(files ${workdir} '*.mha') 6 - 7 -Foreach(file ${files}) • - Set(filename '${workdir}/${file}') • - Set(outfile '${workdir}/out_${file}') 10-SetAppOption(imSeg.inputFilename ${filename}) 11-SetAppOption(imSeg.outputFilename ${outfile}) 12- Run(output ${imSeg}) 13-EndForeach(file)

  10. Condor • Specialized workload management system for compute-intensive jobs • Created by UW-Madison • 15 Years of development • Provides job queueing mechanism, scheduling policy, priority scheme, resource monitoring, and resource management. • Unix and Windows

  11. Condor Universes • Standard Universe (checkpoint) • Vanilla • Parallel Virtual Machine • MPI • Globus • Java (JVM) • Scheduler (no matching)

  12. Condor Example Script Universe = vanilla Executable = EMSegmenter Arguments = ‘brain.mha segmentation.mha --brain’ Output = emsegmentation.out Error = emsegmentation.error Log = emsegmentation.log Requirements = (OpSys =="LINUX" && Arch=="INTEL") Should_transfer_files = YES When_to_transfer_output = ON_EXIT_OR_EVICT Transfer_input_files = brain.mha Queue

  13. Condor Pool Condor Workflow (Vanilla) Central Manager Central Manager Unavailable Execute Machine Dispatch to Central manager = Unavailable Submit job Submit Machine Issue job Busy Workstation Available Execute Machine Central Manager Dispatch to Central manager = Available Execute Machines Idle Workstation Push out job to machine

  14. ClassAd Job ClassAd Match Condor Job Submission Workflow Submit Machine Central Manager Execute Machine 1 Collector Master 3 Master 2 4 Scheduler Negociator 5 Starter

  15. BatchMake Condor Generation

  16. BatchMake Condor Generation • Application/Module description - Inputs/Outputs - Data to transfer to nodes • Loop unrolling • Scope definition • Directed Acyclic Graph (DAG) • Ensure order of execution of jobs

  17. Monitoring Condor Jobs • Flock Monitor (Condor) • Condor Watcher (FLTK) • Condor Web Watcher (Ajax/PHP)

  18. Condor Summary • Pros:- Well supported- (Fairly) easy installation- Complete resources management • Cons:- Script can be complicated- Jobs don’t start right away- Hard to debug

  19. Reporting with BatchMake Developers check-in code CVS/SVN CTest/DART compilessource code Developers review results + MIDASXNat BatchMake validateson datasets Results posted on web(i.e., the batchboard)

  20. Reporting with BatchMake

  21. BatchMake Server • PHP Scripts with MySQL database • Simple C++ HTTP post • Statistical Analysis (Factor analysis: Taguchi). • Dashboard creation capability • User management

  22. BatchBoard Script DashboardHost(http://www.insight-journal.org/batchmake) DashboardUser(‘Julien Jomier') DashboardKey('FKH3BPM1y11VXsMf') CreateExperiment(exp ‘Liver Project‘ ‘Segmentation Experiment' '') CreateMethod(Meth exp 'Registration' 'Register a T1 with noise to a translated T1') AddMethodInput(noiseImageName Meth 'Fixed Image') Set(fixedSlice '${fixedImage}Slice.png') ExtractSlice('${fixedImage}' '${fixedSlice}' 2 1) AddMethodInput(fixedSlice Meth 'Fixed Slice with noise' png) DashboardSend(Meth)

  23. BatchMake and Slicer • BatchMake now distributed with Slicer • Some modules can be run on a collection of data- EMSegment- GaussianBlur • Generic way to run moduleson a collection

  24. BatchMake and MIDAS • STTR Kitware/UNC Martin Styner

  25. Future • Improve packaging • Improve documentation • Improve integration with Slicer • BatchMake and Xnat (discussion)

  26. References • www.batchmake.org • www.insight-journal.org/batchmake