1 / 21

Visualize and Generate MapReduce Logic with Abu: A Simplified Hadoop DSL

In this project, I introduced Abu, a simplified scripting language designed for Hadoop's MapReduce processes. By utilizing a concise notation, Abu aims to streamline the development of MapReduce jobs without excess boilerplate code, allowing developers to focus solely on core logic. It generates Java code, provides visual outputs through Graphviz, and ensures I/O correctness at the domain-specific language (DSL) level. The project is still evolving, with plans for enhancements like flow validation, support for other MapReduce engines, and the incorporation of high-level visualizations.

salome
Télécharger la présentation

Visualize and Generate MapReduce Logic with Abu: A Simplified Hadoop DSL

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. AbuAHadoop Scripting Language & Visualizer Vinod Dinakaran CHUG Oct 21 2010

  2. I started learning Hadoop… Using 2 standard texts…

  3. But it was not until… … that they had this simple notation for the map reduce process:

  4. …scattered through the text they also had….

  5. … both of which seemed like really good ways to represent the process. Which led me to think…

  6. What if I made the nice notation the core, and generate everything else? Visualize Generate

  7. Abu is an implementation of this idea. • Goals: • No boilerplate in the script, just the core MR logic • Still looks like map reduce, i.e., not high level like Pig/Cascade • Generates boilerplate Java, you fill in the method bodies • Generates dot format output so that it can be easily visualized • Analyzes i/o and ensures correctness at DSL level Entirely aspirational notion at this point

  8. A simple example Original Syntax job MaxTemperature: read (LongWritable,Text) from "/path/to/file.ext" using DataReaderClassName mr1 (LongWritable,Text) to ('Text', 'IntWritable') write ('Text', 'IntWritable') to "/path/to/file.ext" using DataWriterClassName mapreduce mr1: map (LongWritable,Text) to ('Text', 'IntWritable') using mapClassname reduce ('Text', 'IntWritable') to ('Text', 'IntWritable') using redClassname Ruby Syntax • job 'MaxTemperature' do • read 'LongWritable','Text','/path/to/file.ext', '' • execute 'max_temp','LongWritable','Text','Text', 'IntWritable' • write 'Text', 'IntWritable', '/path/to/file.ext', '' • end • mapreduce 'max_temp' do • map 'LongWritable','Text','Text', 'IntWritable', '' • reduce 'Text', 'IntWritable','Text', 'IntWritable', '' • end … obviously more simple and complex ones are possible

  9. Demo: Java Code Generation Produces….

  10. … which can be enhanced with the actual method bodies, and other details

  11. … like so

  12. Compile and jar up the code…

  13. .. And run it Todo: Use the tool interface.

  14. Demo: Graphviz Visualization Produces….

  15. That was v0.1

  16. It could do a whole lot more ..and add includes while you’re at it! Make the syntax DRY Add flow validation How about a high level Viz instead of current detailed one? … Or one of a running Job? Maybe I should make it a full DSL – allow definition of map/reduce functions in place using Jruby

  17. .. And be a whole lot better • Refactor Ruby code • Decide on Java implementation • Script the examples from the 2 books to prove out the concept • Script the samples from the Hadoopdistro • Script the standard MR usage patterns (eg. Join) as Abu blocks

  18. Some unintended consequences • Although originally intended as a (personal) learning tool, it could have uses outside of learning • Abstracts away Hadoop interface changes (almost) • Ruby syntax paves way for the possibility of Abu to be a true DSL • Visualizing a defined job led to the idea of visualizing a running one • With modifications, the design could even support other MR engines

  19. Similar Projects Jruby on Hadoop: http://github.com/fujibee/jruby-on-hadoop Papyrus: A full fledged Ruby DSL for Hadoop http://github.com/fujibee/hadoop-papyrus

  20. Thanks! • Interested? • Join me or fork away : http://github.com/vinodkd/abu • Vinod.dinakaran@gmail.com • Vinodkumar.dinakaran@orbitz.com

More Related