Download
progress report n.
Skip this Video
Loading SlideShow in 5 Seconds..
Progress report PowerPoint Presentation
Download Presentation
Progress report

Progress report

113 Vues Download Presentation
Télécharger la présentation

Progress report

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

  1. Progress report 2012/05/01 匡建慈

  2. Reminding • Goals : to build a multi-core platform with Hadoop environment Application layer User program User API Tegra2 Transaction layer Linux Device Driver Base board & FPGA& fiber ports DMA in Tegra2 Router circuit in FPGA Fiber link Physical Layer

  3. What to do and what we have • What we have • Tegra2 software program co-work with FPGA to do simple work . • A circuit example with fiber communication • What to do • a example by combining examples we have to achieve a framework from software , hardware , and fibers . ( O ) • Example interface unified • Achieve API package • Interrupt mechanism • Porting Hadoop environment on proposed framework . ( X )

  4. Combined example block diagram Fiber link Circuit layer Driver Software Clk_wiz GTP BASE DMA Linux Device Driver User Prog. TX_IF TX_Buf DMAIF RX_IF RX_Buf BASE_CLK clk_120m

  5. API package • Global variable : device number , buffer pointer • init : open device , memory mapping • Finalize : close device , memory um-mapping . • FPGAWrite : write buf_w into circuit layer by DMA • FPGARead : Read data from circuit by DMA and store into buf_r • header packet mechanism • In future , it wiil be applied on APIs

  6. Interrupt mechanism • Circuit layer : • pull up MEM_R/W_Ready to setup an interrupt to CPU • A interrupt means it’s ready for the next user read/write call • Driver layer • Driver code in original release might be got stuck due to mutex usage . • According to previous work on MultiARM, rewrite driver by a kthread interrupt handler and two wait_events 25 ns

  7. Result

  8. 2-node test • A single SendRecv program running on 2 nodes • Add a ‘!’until the length is larger than specified value Receiver Sender

  9. Next • Try to setup Hadoop environment on proposed framework . • Do on my laptop • Setup JAVA on MaCube • Setup Hadoop environment on MaCube • Modify the calling interface to my APIs

  10. Progress report 2012/05/15 boy13

  11. Last week • Completed • Setup programming model and achieve framework covered with software , hardware , GTPs . • Combining examples • API package • Interrupt mechanism • To do • Setup Hadoop on proposed framework • Setting Hadoop on this laptop O • Setting JAVA Runtime Environment on MaCubeO • Setting Hadoop environment on MaCubeX • Modify calling interface by my APIs X

  12. Setting Hadoop on my laptop • Single node setting • Download Hadoop and jdk and extract them . • <Hadoop Path>/conf/hadoop_env.sh • Export JAVA_HOME=<JdkPath> • <Hadoop Path>/conf/core-site.xml • hadoop.tmp.dir <a path to put /tmp> • fs.defailt.name hdfs://localhost:9000 • Dfs.replication 1 • <Hadoop Path>/conf/mapred-site.xml • Mapred.job.tracker localhost:9001 • More on http://changyy.pixnet.net/blog/post/25245658

  13. Hadoop example execution • HDFS format • <Hadoop Path>/bin/hadoopnamenode -format • Start and stop • <Hadoop Path>/bin/start-all.sh • <Hadoop Path>/bin/stop-all.sh • HDFS operation • <Hadoop Path>/bin/hadoopdfs –mkdir input • <Hadoop Path>/bin/hadoopdfs –ls • <Hadoop Path>/bin/hadoopdfs –put <filepath> input

  14. Hadoop example wordcount execution • <Hadoop Path>/bin/hadoop jar hadoop-0.20.1-examples.jar wordcount input output • See execution result • <Hadoop Path>/bin/hadoopdfs –cat output/*

  15. Multi-node setting • Hosts • /etc/hosts : A table record machine name and its IP • /etc/hostname : machine name • ssh connections • sudo apt-get install openssh-server • <Hadoop Path>/conf • slaves : machine name of data-nodes • masters : machine name of name-node • start-all.sh

  16. After start-all.sh • <Jdk Path>/bin/jps • <Hadoop Path>/bin/hadoopdfsadmin –report • More on : http://changyy.pixnet.net/blog/post/25612440

  17. Setting JAVA Runtime Environment

  18. cp /media/sda1/ejre1.6.0_32.tar.gz ./ • tar -xjf ejre1.6.0_32.tar.gz • export PATH=/home/root/Desktop/ejre1.6.0_32/bin:$PATH

  19. Next to do • Try to setup Hadoop environment on proposed framework . • Do on my laptop • Setup JAVA on MaCube • Setup Hadoop environment on MaCube • Modify the calling interface to my APIs

  20. Progress report 2010/05/29 boy13

  21. Recently • Goals : to build a multi-core system with hadoop environment • Work items • Setup programming model and achieve framework covered with software , hardware , GTPs . • Combining examples O • API package O • Interrupt mechanism O • Router mechanism • Setup Hadoop on proposed framework • Setup up on my laptop O • Setting JAVA Runtime Environment on MaCube O • Setting Hadoop environment on MaCube

  22. Router mechanism • A Light-Weight Parallel Router for FPGA-based Networks-on-Chip • Department of ECECS , University of Cincinnati, 2005 Application layer User program User API Transaction layer Linux Device Driver Physical Layer DMA in Tegra2 Router circuit in FPGA Fiber link

  23. Router mechanism • 2-D mesh architecture • Routing mechanism • XY routing • Port • Input Channel • FIFO interface • Be requested from Output Channel of a port in another router • Request to X-bar switch to access Output Channel • Output Channel • Crossbar Switch

  24. For out architecture • 2-D mesh => 3-D mesh • Port => Local Port , Fiber Port • Local Port : interface for FPGA and DMA in tegra2 • Fiber Port : interface for fiber link in proposed framework Tegra2 Base board & FPGA& fiber ports

  25. Setting hadoop in proposed framework • Setting up namenode • applet not found • What is applet ? • JRE v.s. Java Applet • By several weeks asking ,I conclude that JAVA Applet isn’t supported in embedded JAVA .

  26. Setting up datanode in MaCube • Laptop plays namenode role in this work . • Setting ssh connection • Pre-requite : openssl , zlib • Download source package • Extract it to local directory • ./configure –help • ./configure <options for cross compile> • Modify makefile manually if necessary • make && make install

  27. Error occur when I key make install for openssh • Source pack : 5.7 , 5.9 • Toolchain : arm-2009q3 , cc.2011.03

  28. Next • Router mechanism • Complete design for Local port , Crossbar switch • Change local port with fiber port • Complete 2x2 2-D mesh architecture • Setup hadoop environment • Build openssh and setup connection • Try datanodestart service

  29. Progress report 2012/06/19 boy13

  30. Recently • Goals : to build a multi-core system withouthadoop environment • Work items • Setup programming model and achieve framework covered with software , hardware , GTPs . • Combining examples O • API package O • Interrupt mechanism O • Router mechanism O • Multi-core system • Setup Hadoop on proposed framework • Setup up on my laptop O • Setting JAVA Runtime Environment on MaCube O • Setting Hadoop environment on MaCube X • Namenode software service

  31. Router mechanism • Router • Input Channel , Output Channel • Input Request mechanism , Round-Robin Arbitrator • Parallel Router by cross-bar switch • Testing pattern • Local port O • Local port + Router O • Local port + Router + 1 fiber port O • Local port + Router + 4 fiber port O

  32. DMA id Local port Router East port West port North port South port GTP_J4 GTP_J5

  33. Multi-core system • So far , only 1-1 send-recv pairs , and no packet overlapping occur . • Read mechanism in driver should be designed • When read interrupt comes • configure DMA and put data into a buffer • Wakeup kthread (interrupt handler) • Kthread • Put data in buffer in a list kept in driver • Read call • Search list in driver • If found , copy data into read buffer API allocated • Otherwise , sleep • Re-check circuit design • Whether data not sent but be replaced or not • Buffer empty register

  34. Setting hadoop environment • Hadoop setting • Download hadoop source and JRE into MaCube • Modify <hadoop path>/conf/ * .xml • Build ssh connection • ./start-all.sh • ./bin/hadoop <execution commands> • 2 use-case • MaCube play both namenode and datanode roles • MaCube plays datanode role and laptop/PC act as namenode

  35. Problem encountered for both datanode and namenode role • bash : no such file or directory • What is bash ? • ln –s /bin/sh /bin/bash • bash : applet not found

  36. By several weeks googling and asking , I conclude that JAVA Applet isn’t supported in embedded JAVA . • Taiwan Hadoop Forum • http://hadoop.nchc.org.tw/phpbb/index.php?sid=021ae10a0e0609ab818b3e7062aaa314 • Embedded JAVA , no bash => no applet => hadoop can’t be built . • So ……

  37. Namenode software service • Motivation : hadoop is not friendly with embedded system . • We build a simple software service of cloud system • A simple service to submit job • Package source code , data , and execution arguments • cross-compile source code • Send package to datanode scheduled • Result returned • Graphics user interface and it’s easy to use. • Embedded JAVA !! • New thread and system call to execute program received • Simple distributed file system • Connect with student of Prof. Shiao

  38. To do • achieve multi-core system hardware framework • Complete read mechanism • Put fiber port from 4 to 6 • Design namenode software service • A quick job submit flow • Easy to use and graphical

  39. Progress report 2012/07/10 boy13

  40. Recently • Goals : to build a multi-core system • Work items • Setup programming model and achieve framework covered with software , hardware , GTPs . • Combining examples O • API package O • Interrupt mechanism O • Router mechanism O • Multi-core system • Setup Hadoop on proposed framework • Setup up on my laptop O • Setting JAVA Runtime Environment on MaCube O • Setting Hadoop environment on MaCube X • Namenode software service • Demo items

  41. Multi-core system • Transportation testing • X-axis transportation • Y-axis transportation • 2-D mesh transportation 0 1 2 3 0 1 2 3

  42. Multi-core system • Serial transportation • Motivation : sometimes a byte data shift

  43. Multi-core system • Serial transportation problem • Reference clock skewing • Solutions : • Sink : make the initiator and target correct the timing when initial the transaction • Re-send : if data check is error by some mechanism , then request initiator to resend • The author of GTP_EXAMPLE says , it’s locked when the first sink , and can’t be unlocked . • => only can try re-send method .

  44. Method thinking • do something when error has checked at the first time . • Ack/Nak mechanism . • Deadlock possible ? GTP CORE GTP CORE TX_IF TX_IF RX_IF RX_IF

  45. Namenode software service • TCP communication • Try : a simple chat room program • Shell program

  46. Server Client Initial socket Initial socket Get connection Create thread to message passing Listen client Check queue is empty Create receive thread Take mutex , pop queue , Socket send Create thread to recv Message recv Socket recv chat Take mutex and queue it Print

  47. Demo items • Demo item 1 : • Using Ethernet to do Map-Reduce or SPCP • Demo item 2 (possible) : • Using Ethernet to transport program and data • Using fiber to communication with components 1 2 3 4 5 6 7 8

  48. Progress report 2012/08/01