PRISM: Planning a Center for P erceptual R obotics, I ntelligent S ensors & M achines at the City College of N

1. C CVC L PRISM: Planning a Center forPerceptual Robotics, Intelligent Sensors & Machinesat the City College of New York Zhigang Zhu Visual Computing Laboratory Department of Computer Science City College of the City University New York http://www-cs. ccny.cuny.edu/~zhu/ John (Jizhong) Xiao Robotics and Intelligent Systems Lab Department of Electrical Engineering City College of the City University New York Website: http://www-ee.ccny.cuny.edu/www/web/jxiao/jxiao.html

2. Outline • Who Are We & Project Goal • Problems and Promises • Research Theme/Threads • Team and Faculty Involvement • Infrastructure • Education and Outreach • Academic Outreach / Partnership • Evaluation Plan • Summary NSF Infrastructure 2005

3. Outline • Who Are We & Project Goal • Problems and Promises • Research Theme/Threads • Team and Faculty Involvement • Infrastructure • Education and Outreach • Academic Outreach / Partnership • Evaluation Plan • Summary NSF Infrastructure 2005

4. CUNY System w/ Vision and Robotics CUNY City College Graduate Center Brooklyn Hunter Queens … Flagship campus HSI / MI (PhD granting institution*) SOE * Site of PhD programs in Engineering ME CS EE Planned Center for Perceptual Robotics, Intelligent Sensors and Machines under NSF MII planning project Grant No. CNS-0424593 PRISM Center NSF Infrastructure 2005

5. Outline • Who Are We & Project Goal • Problems and Promises • Research Theme/Threads • Team and Faculty Involvement • Infrastructure • Education and Outreach • Academic Outreach / Partnership • Evaluation Plan • Summary NSF Infrastructure 2005

6. Research Theme and Threads • One Research Theme: RISE-NET • Robotized Intelligent SEnsor NETworks in a 3D space • w/ air, ground, wall-climbing robots • Four Research Threads • Perceptual Robots – navigate, network & plan • For surveillance & inspection • Intelligent Sensors – multimodal, multi-level • For surveillance, inspection & education • Smart Machines – HCI - M.I and H.I. • For human interfaces in surveillance & advanced e-learning • Reliable Communication • Ad hoc networks, Multimedia networks • More Detailed Technical Experiences • Pervasive Computing Workshop NSF Infrastructure 2005

7. Thread 1Perceptual Robotics:Mobile Sensor Network in 3Dfor surveillance and security PRISM Center • Joint Work with • George Wolberg, John Xiao @ CCNY • Bob Haralick @ CUNY Graduate Center • Ed Riseman, Al Hanson, Rod Grupen @ UMass • Tom Huang @ UIUC • Supported by NSF, DARPA,ARO, NYSIA, ACTI, etc NSF Infrastructure 2005

8. Goal • Dream: • transform the present 2-D world of mobile rovers into a new 3-D universe –air, ground, under-object, and wall-climbing • move on ground, climb walls, walk on ceilings, transit between surfaces. • Applications: • Urban warfare applications: surveillance and reconnaissance, weapon delivery, guiding perimeter around a building, etc • Security and counter-terrorist applications: intelligence gathering about a hostile situation within a building, etc. • Inspection and maintenance applications: routine inspection of buildings, nuclear containment domes, and other hard-to-reach places, inspection of aircraft, sand blasting of ship hulls, etc. • Other Civilian applications: assistance in firefighting, search and rescue operations, etc. NSF Infrastructure 2005

9. Some Real Examples • City/Urban modeling (NSF/AFRL/NYSIA) - airborne video: far-view (1000 ft) • Robot Navigation (DARPA, ARO)- ground video: medium range (~100ft) • Facade inspection (ARO) – climbing robots: near-view (inches?) • Under-vehicle inspection (ACTI) - car drives over camera: near-view (< 8 in) NSF Infrastructure 2005

10. Objectives • Rapidly create large FOV image mosaics that are • geo-referenced, • with 3D (stereo) viewing and • with all dynamic targets identified • As a light UAV flights over an area NSF Infrastructure 2005

11. Region-Based Stereo Matching Left mosaic – Full Depth Map NSF Infrastructure 2005

12. Major Robots Acquired/Fabricated • Ground Robots at CCNY • Wall-Climbing Robots at CCNY NSF Infrastructure 2005

13. Robot Localization and View Planning • Mutual awareness • Heterogeneous sensors • Active Sensing • View Planning • for tracking and recognition NSF Infrastructure 2005

14. Thread 2Intelligent Sensors:Multimodal Human Signatures for Surveillance & Inspection PRISM Center • Joint Work with • Weihong Li, George Wolberg (CCNY) • Ning Xiang (RPI), Tom Huang (UIUC) • Lt. Jonathan Lee, Mr. Robert Lee (AFRL) • Supported by AFRL, CUNY RF • Publications : OTCBVS05 NSF Infrastructure 2005

15. Figure 1. Multimodal remote hearing by human and machine: system diagram Multimodal Human Signatures • Collaboration with • Prof. Tom Huang at UIUC • Prof. George Wolberg • Goal • Remote hearing by human and machines • Challenges • Noisy LDV signals • Low-res, non-front faces • Target detection?? NSF Infrastructure 2005

16. Major Sensor Acquisition • Polytec Laser Doppler Vibrometer (LDV) • OFV-505 Sensor Head, OFV-5000 Controller • Tip-Tilt Stage VIB-A-P05 with Telescope • FLIR ThermoVision Camera • ThermoVision® A40M Infrared Camera • Canon Color/IR PTZ Camera • Canon VC-C50i Low Light IR PTZ Camera • Other video cameras • Omnidirectional cameras, stereo head • Camcorders, webcam, CMOS sensors NSF Infrastructure 2005

17. Thread 3Smart Machines: Multimedia Integration in a Virtual Classroom PRISM Center • Collaborators • Rick Adrion, Ed Riseman, Al Hanson, Jim Kurose, (UMass) • Parviz Kermani, IBM / PINY • Student: Weihong Li, Hao Tang, Chad McKittrick • Supported by • CUNY CISDD NSF Infrastructure 2005

18. Multimedia Virtualized Classroom • Goal: automating e-learning in MVC • Automatic capture (A, V, WB, BB) • Intelligent media integration • User-customized presentation • Problem and Promise • Large proposal with UMass declined by NSF CRI • Keep Trying… NSDL Collaborative Research • with UMass who has a successful VIP on campus • MII -> CRI, more competitive NSF Infrastructure 2005

19. conversion index PPT Capture Add-on registration Mimio Virtual Ink video transformation Remote Reality Omnidirectional Camera VC: Automatic MM capture NSF Infrastructure 2005

20. Active WWW link Synthetic Hi-Res digital display Slide combined with video Panoramic mosaic from video Annotated and active instructor VC: Media Integration Toward an immersive VC interface Question Window, ToC, etc NSF Infrastructure 2005

21. VC: Media Integration Instructor Extraction…and integration video image shadow contour NSF Infrastructure 2005

22. Outline • Who Are We & Project Goal • Problems and Promises • Research Theme/Threads • Team and Faculty Involvement • Infrastructure • Education and Outreach • Academic Outreach / Partnership • Evaluation Plan • Summary NSF Infrastructure 2005

23. PRISM Team (15+ people in CS, EE &ME) • Prof. Myung Lee (EE) • Wireless sensornetworks, ad hoc networks, Multimedia Networks • Prof. Esther Levin (CS) • Spoken languageHCI, machine learning • Prof. Ali Sadegh (ME) • MEMS, micro device, biomechanics body/machine interaction • Prof. George Wolberg (CS) • Vision, graphics • Prof. Jizhong (John) Xiao (EE)* • Robotics, control, sensor network • Prof. Zhigang Zhu (CS)* • Vision,multimodal sensors, HCI * Co-PI of the MII project NSF Infrastructure 2005

24. Faculty Involvement • Problem • MII/CRI limited funding for faculty support • Heavy teaching load at CCNY • Lack interest in student mentoring work • Center allocates funding for small focused projects • Call for proposals from mentoring faculty • Funding for faculty and student research* • Obtain course release time (commitment from CCNY) • Seek External Funding (NSF, DoD, etc) NSF Infrastructure 2005

25. Current Funded Projects • Zhu, Wolberg, Haralick, Dynamic Pushbroom Stereo Mosaics for 3D and Moving Target Extraction. 02/15/05- 02/14/09, $267,437 (AFRL) • Xiao, Zhu, Cooperative Wall-climbing Robots in 3-D Environment for Surveillance and Target Tracking, $375,237, 11/01/04-10/31/07 (ARO) • Uyar, Xiao, Smart Reconfigurable Miniature Robot System Based on System-On-Programmable-Chip Technology”, $202,844, 09/04 –09/06 (NSF MRI) • Zhu, Integration of Laser Vibrometry with Infrared Video for Multimedia Surveillance Displays, $94,076, 08/24/03 – 10/24/04 (AFRL) NSF Infrastructure 2005

26. Current Funded Projects • Wolberg, Log-Polar Transforms for Optical Image Processing and Target Recognition, $600,000, 3/1/03 -10/1/06 (ONR) • Lee (Co-PI), CTA ( Collaborative Technology Alliance), Wireless ad hoc networks, routing, MAC, multicasting, reliable transport, $2.5 M, 07/01-05/06 (ARL) • Lee (Director), Samung-CCNY Joint Laboratoryon wireless sensor networks ,2003-07 $1.2M (Industry) • Sadegh, Northrop Gruman, $10K, Crown Corp., $52K, ALCOA Inc., $14K, Con Edison $8.5K. (Industry) NSF Infrastructure 2005

27. Current Funded Projects • Zhu, Tracking Moving Objects via a PTZ or Moving Camera, $23,333, 08/01/04- 07/31/05 (NYSIA) • Zhu, Wolberg, Integration of Laser Vibrometry, Infrared and Video for Multimodal Human Detection, $50,000, 02/19/04-02/18/05 (CUNY Research Equipment Grant) • Xiao, Lee, Acquisition of equipment for multi-robot and mobile sensor networks research, $22,250, 7/05~7/06 (CUNY Research Equipment Grant) NSF Infrastructure 2005

28. Pending Proposals to NSF… • Zhu, Wolberg, Huang (UIUC), Multimodal Remote Hearing for Humans and Machines, , $731, 776.00, 01/0/1/06-12/31/08 (NSF IIS) • Zhu, Levin, Adrion & Riseman (UMass), Transforming Digital Video Libraries for Intelligent Access and Navigation, $240,499.00 (NSF NSDL) • Levin, 2 NSF IIS proposals • Grossberg, NSF CAREER proposal • …… • NSF CRI proposal in progress • MII planning award ->CRI project NSF Infrastructure 2005

29. Summary: Funding for Faulty Research • Overall > $5 M in the last 5 years • Vision + Robotics > $1.6 M in the last 3 years • New faculty involvement • Active small-scale collaborations • SOE at CCNY investment • Physical infrastructure, new science center, release time, lecture series, very attractive startups • => A Center for multidisciplinary, large-scale collaborations in research and education NSF Infrastructure 2005

30. Outline • Who Are We & Project Goal • Problems and Promises • Research Theme/Threads • Team and Faculty Involvement • Infrastructure • Education and Outreach • Academic Outreach / Partnership • Evaluation Plan • Summary NSF Infrastructure 2005

31. Infrastructure • Existing Equipment • As described with the research threads • Wish List for CRI 2006 • Airborne and Outdoor Mobile Platforms • UAVs, outdoor robots • Heterogeneous Sensor Packages • Miniature and high-end cameras, acoustic • GPS, INS , ladars, onboard processors • Various Levels of Vision Processors • Compact, high-end, low-cost, low-power • Wireless Sensor Networks • Practical for imaging sensors and robots NSF Infrastructure 2005

32. Outline • Who Are We & Project Goal • Problems and Promises • Research Theme/Threads • Team and Faculty Involvement • Infrastructure • Education and Outreach • Academic Outreach / Partnership • Evaluation Plan • Summary NSF Infrastructure 2005

33. Education and Outreach • CCNY is both MI and HSI • Student retention • New courses and academic programs • Undergrad, MS and PhD • Outreach to K-12 students • Complete the pipeline NSF Infrastructure 2005

34. Education and Outreach (I) • CCNY is MI and HSI • 1/3 African Americans, 1/3 Hispanic-Americans • Urban institution without dorms • Long distance commuters • Student faces financial hurdles • Majority of students hold part/full-time jobs • MAJOR challenge at CCNY • Students retention • Solution ? • attractive research and education programs NSF Infrastructure 2005

35. Education and Outreach (II) • Introduce new courses • video computing, mobile sensor network • Reshape capstone design courses • Combine robotics, vision and mechatronics • Create “Pipeline” to • attract high school students to CISE disciplines • Retain and timely graduate under-rep undergrads • Support talented minority students for graduate ed. • All team members are on the newly established CpE program NSF Infrastructure 2005

36. Education and Outreach (III) • NYC-LSAMP • Louis Stokes Alliance for Minority Participation • New York Academy of Sciences (NYAS) • Summer Research Training Program (SRTP) • New York/New Jersey FIRST • For Inspiration and Recognition of Science and Technology • Mentor high-school kids participating FIRST robotics competitions • Organize workshops • CCNY Robotics Club (organize outreach activities) • Climbing Robot Competition 2005 in UK • Affinity Mentoring NSF Infrastructure 2005

37. Outline • Who Are We & Project Goal • Problems and Promises • Research Theme/Threads • Team and Faculty Involvement • Infrastructure • Education and Outreach • Academic Outreach / Partnership • Evaluation Plan • Summary NSF Infrastructure 2005

38. Academic Outreach / Partnership • Advisory Committee • of experts in robotics, computer vision and sensor net • Lecture Series • on computer vision, robotics and HCI • Visits • Institutions with successful PRISM programs • MI/HSI institutions NSF Infrastructure 2005

39. Advisory Committee • Roderic A. Grupen • Director, Laboratory for Perceptual Robotics, UMass-Amherst • Thomas. S. Huang • Chair of HCII, Head of IFP, Beckman Institute, UIUC • Allen R. Hanson and Edward Riseman • Co-Directors, Computer Vision Laboratory, UMass-Amherst • Richard Volyes • Director, Collaborative Systems Laboratory, UMinn • Ning Xi • Director, The Laboratory of Robotics and Automation, MSU • USC, UCSD, UCF, CU… NSF Infrastructure 2005

40. Lecture Series • Web page • http://www-cs.ccny.cuny.edu/ ~zhu/CcvcL/html/ccny_lecture_series.html • Objectives • excite interests, foster collaboration & increase visibility • Lectures given • Huang at UIUC, Hanson/Riseman & Grupen at UMass, Shah at UCF • Wittenburg at MERL, Kumar at Sarnoff ( CUNY GC events) • Lectures in plan • Ning at MSU, Trivedi at UCSD , Medioni/Nevatia at USC • and more NSF Infrastructure 2005

41. Visits to other institutions • Visits Made • USC IRIS and IMSC (vision, robotics and graphics labs) • Presentation by Z. Zhu • UCSD Robotics and Computer Vision Lab • UMass Computer Vision Lab and Lab for Perceptual Robotics • Presentations by Z. Zhu and J. Xiao • In Plan • MERL (August for given a talk) • CMU Robotics Institute (Fall) • UPenn GRPASP Lab (Fall) • U. Texas at El Paso(Fall) • MSU, Uminn… NSF Infrastructure 2005

42. Outcomes of Academic Outreach • All the three objectives –pleasant surprise • excite interests, foster collaboration & increase visibility • Faculty and students involvement • Recognition by outside world • Multiple Joint Proposals • UIUC, UMass, RPI • USC, UT, etc in plan NSF Infrastructure 2005

43. Outline • Who Are We & Project Goal • Problems and Promises • Research Theme/Threads • Team and Faculty Involvement • Infrastructure • Education and Outreach • Academic Outreach / Partnership • Evaluation Plan • Summary NSF Infrastructure 2005

44. Evaluation Plan • Research achievement metrics and • Educational effectiveness metrics • based on the corresponding NSF guidelines • Possible metrics • immediately tangible research results seeded by the funding (e.g., student participation, publications) • improvement of minority students’ retention rates • number of students entering graduate schools, etc • Under development… NSF Infrastructure 2005

45. Outline • Who Are We & Project Goal • Problems and Promises • Research Theme/Threads • Team and Faculty Involvement (*) • Infrastructure • Education and Outreach (*) • Academic Outreach / Partnership • Evaluation Plan • Summary NSF Infrastructure 2005

46. Summary • Conclusions Bases • internal examination and feedback from external committee members • Recognition in Research and Education Programs • confident in excelling our research and innovating the minority educational programs under the theme of the PRISM center: RISE-NET • Hardware infrastructure • is critical for both research and ed., and this is well recognized • The most challenging issue • to find mechanisms (funding!) to motivate talented researchers and educators to devote into PRISM center • Goal - to make CCNY • a center of excellence in cutting-edge research and • a national urban model for minority education • in the fields of robotics, computer vision and sensor network NSF Infrastructure 2005

47. Thank you! http://www-cs. ccny.cuny.edu/~zhu/ http://www-ee.ccny.cuny.edu/www/web/jxiao/jxiao.html NSF Infrastructure 2005