PhD Prospectus Presentation LTC Edward P. Mattison

1. PhD Prospectus PresentationLTC Edward P. Mattison COMPANION: A Cooperative Mapping and Adaptive Indoor Navigation System utilizing Aerial and Ground Robotic Vehicles Date: Thursday, October 23rd, 2008 Time: 1:00 pm Location: G-pod Conference Room Watson Engineering Bldg Binghamton, University

2. Agenda • Introduction • Project Summary and Purpose • Possible System Applications • Principal Elements of the System • Existing work in the field • Unique Aspects of the Project • Explanation of Phases: I-V • Potential Contributions of Project • Projected Timeline • Q&A and Conclusion

3. Introduction

4. Edward P. Mattison Position: • Lieutenant Colonel, U.S. Army • Information Systems Officer • Assistant Professor, West Point Education: • B.S., Comp. Sci., West Point, 1991 • M.S., Comp. Sci., BU, 2001 • PhD Candidate, BU, 2006-TBD

5. COMPANION Principals • LTC Edward Mattison, Primary Researcher, Doctoral Candidate • Dr. Kanad Ghose, Dissertation Supervisor and Committee Chair: Systems, Architecture, Power • Dr. Patrick Madden, Committee Member: Algorithms • Dr. Les Lander, Committee Member: Java Programming, OOP Design • COL Bryan Goda, Committee Member: Outside Examiner, Reconfigurable Computing • Mihai Puscasu, Assistant Researcher, Master’s Candidate

6. Project summary

7. Project Motivation and Approach • Lightweight, small form-factor airships (blimps) can be very useful as sensing/surveillance platforms • Less expensive than fixed or rotary wing UAV’s or micro-air vehicles • Airships are much more robust • Ideal for applications indoors and confined areas • Challenges: • Limited payload, computing power, and storage space • Limited on-board power source • Indoor operation precludes the use of GPS • Our approach: cooperative and adaptive operation of blimp and autonomous ground vehicle • Demonstrate fully-functional prototype of system

8. Proposed System • COMPANION: A Cooperative Mapping and Adaptive Indoor Navigation System utilizing Aerial and Ground Robotic Vehicles • System of cooperating autonomous vehicles, one of which is an airship • General techniques developed within this project are applicable to a wider variety of vehicles

9. System Architecture ANS MBS HUMAN AGE AGE AGE

10. System Overview • All system components cooperate/coordinate with one another • All vehicles within system have adaptive roles • Roles change dynamically based on certain conditions (power constraints, computational complexity, etc.) • Control techniques vary across a wide spectrum from human remote operation to fully autonomous operation Dissertation contribution: • a set of dynamically adaptive algorithms that facilitate the coordination and cooperation of heterogeneous autonomous vehicles

11. Possible System Applications • Indoor military reconnaissance • Search and rescue operations • Information gathering during hostage crisis • Environmental monitoring • Hazardous material situational sampling • Routine building monitoring • Crowd surveillance and major events

12. Principal components of companion system

13. Mobile Base Station (MBS) • Acts as System Master • Hosts DACA algorithm • Repository for the common system map • Issues all route instructions and other commands • Built aboard an L2Bot platform • Two drive motors with low-cost motor controllers • Core 2 Duo laptop handles all processing • Bluetooth and 802.11g networking • Ultrasonic sensors to “see” walls and obstacles • Webcam and IR Seeker for “vision” capabilities. • Platform is currently operated telemetrically

14. MBS Advantages/Disadvantages Advantages: • Substantial computing power (Core 2 Duo) • Substantial storage capacity (250 GB hard drive) • Substantial battery power (operates 2+ hours moving) • Can be remotely operated by Wi-Fi or Bluetooth Disadvantages: • Limited communications range to explorers (Bluetooth) • Larger size makes obstacle avoidance difficult • Produces significant noise during movement

15. Autonomous Ground Explorer (AGE) • Acts as Slave System • Autonomously Navigates • Use Reactive Navigation and Opportunistic Localization • Observes its immediate environment • Navigates down hallways and corridors • Avoids walls and obstacles • Follows routing commands given by MBS • Determines location from odometry, images and sensor data • Will maintain partial maps on-board and provide periodic updates to the MBS • Based on the Mindstorm NXT platform. • Consist of two drive motors, three ultrasonic sensors, a wireless camera, and an IR Beacon

16. AGE Advantages/Disadvantages Advantages: • Autonomously navigates hallways • Small size increases maneuverability and stealth • Relatively quiet operation Disadvantages: • Limited computing power (ARM7 Processor) • Extremely limited storage capacity (256 KB) • Limited battery power (operates less than 1 hour) • Limited communication distance to MBS (Bluetooth)

17. Aerial Navigation System (ANS) • Serves as “Eye-in-the-Sky” • Navigates telemetrically or autonomously • Offers a unique viewpoint for capturing additional images and relevant information not attainable from the ground • Downward and forward facing cameras provide valuable look-ahead information to ground explorers • Platform resides on a five foot, helium-filled, airship • This system has three directional propellers which provide six degrees of freedom in its movement • On-board processing done by autopilot controller board • Infrared seeker allows it to follow ground explorer

18. ANS Advantages/Disadvantages Advantages: • Airship has long communication range (digital radio modem) • Airship is a very cost-effective platform • Airship is extremely quiet/visually blends with environment • Airship moves slowly (provides almost hover capability) • Can bump objects without damage/no hazard to people • Good processing power (150 mips RISC processor) Disadvantages: • Airship has extremely limited payload capability • Airship has limited battery power (less than 1 hour) • Platform lacks accurate odometry to measure movement • Limited storage capacity (16 Mb)

19. Sensing Capabilities • Ultrasonic Sensors: measure distance to walls and obstacles • Wireless Cameras: capture still images and video to record landmarks and verify location • Infrared Sensor: identifies the direction and intensity of infrared light to allow “following” of other platforms Images Processing at MBS Preprocessing AGE & ANS Ultrasonic Infrared

20. Existing work

21. Existing Work • Fixed infrastructure navigation • Utilizes fixed beacon landmarks • Usage limited to mostly industrial automation • SLAM (Simultaneous Localization and Mapping) • Absolute navigation technique • Computationally intensive and storage hungry • Ideally suited to outdoor navigation • Potentially infinite landmarks and map details • A plethora of variations exist • Most rely heavily on odometry and landmarks

22. Existing Work Examples (Cont.) • Multi-Vehicle navigation and mapping • Homogenous groups of vehicles • Adaptability is non-existent • Mining Vehicle reactive navigation • Single vehicle system • Mine topology is very limited (intersections and tunnels) • No existing work that exploits the capabilities of blimps and addresses their inherent capabilities and challenges

23. Unique aspects of companion

24. Adaptive Control Algorithm Dynamically Adaptive Coordination Algorithm (DACA) • The DACA is the algorithm at the heart of the systems’ ability to adapt vehicle roles to meet to changing conditions • As conditions change, the role of each system will change. The algorithm will monitor the overall system in real-time and dynamically adapt the system roles • Three or more systems will be continuously in communication with base station • This is a very complex system. Effectively managing this complexity is our main objective

25. Dimensions of Adaptability • Types of functions performed at the MBS versus those performed on explorer systems • Frequency and extent of data exchange between the MBS and explorer systems • Resolution and sampling rate of sensor data • Location and quality of map data stored at MBS and explorer systems

26. Adaptation Triggers • Battery power remaining at each system • Complexity of current scene/neighborhood • Abrupt occurrence or obstacle • Storage availability at remote systems • Current response time at each system • Goal is to optimize all three dimensions POWER STORAGE PERFORMANCE

27. Indoor-Only Focus Advantages: • Indoor pathways are usually arranged in a rectangular, regular pattern • Building interiors often contain few obstacles Disadvantages: • Steel and concrete construction restricts communications ranges • Indoor environment prohibits the use of GPS navigation systems and compass devices • Building interiors often lack significant “landmarks” found outside

28. Hybrid Mapping and Navigation • COMPANION system will utilize a topological graph to represent the map • Hallways, intersections, and rooms can be represented by a simple undirected graph • Nodes of the graph represent intersections, doorways, landmarks, etc. • Edges of the graph represent the pathways, hallways, and corridors between the nodes • Metadata will be added to both the nodes and the edges to assist in localization as well as navigation • The MBS can issue routing commands as simple paths through a graph

29. Topological Map Representation (a) Building Map (b) Topological Graph

30. Absolute vs. Relative Navigation • SLAM • Absolute navigation technique • Continuous localization • Computationally intensive and storage-hungry • Ideally suited to outdoor navigation • Potentially infinite landmarks and map details • COMPANION • Relative navigation technique • Opportunistic localization • Computationally moderate and storage-friendly • Ideally suited to indoor navigation • Manageable landmarks and map metadata

31. Heterogeneous Aerial and Ground • COMPANION uses three or more platforms with varying capabilities • Mobile Base Station • Autonomous Ground Explorer • Aerial Navigation System “eye-in-the-sky” • Platforms can be added to the system while in operation • The system emphasizes platform advantages (aerial images from airship) • The system minimizes platform limitations (only pre-processing of sensor data on explorers) • Multiple points of reference offer increased situational awareness

32. Spectrum of Control

33. Explanation of Phases 1-v

34. Phase I Consists of the mobile base station and one ground explorer robot • The base station is telemetrically controlled by a human operator • The base station has a camera ,three ultrasonic sensors to aid navigation • The base station stores a “pre-configured” map of the building • The explorer vehicle is autonomous in a Reactive Navigation mode • The explorer “sees” its surroundings with a camera and three US sensors • The explorer “follows” the route given by the base station • The explorer reports information along the route (intersections/obstacles) • The base station can change the route based on explorer feedback • No new mapping at this phase, only updates are made on path availability • The explorer must stay within communications range of the base station

35. Phase II Consists of the mobile base station and one ground explorer robot • The base station remains telemetrically controlled • The explorer vehicle is autonomous between map nodes • The system starts with a base map, but adds the ability to update • Map changes: additions, deletions, adding of graph metadata • Metadata is limited to odometry data for graph edges • Base station monitors the battery level of the explorer vehicle • The base station can dynamically change the explorer path or stop its mission based on power

36. Phase III Consists of mobile base, the aerial “eye-in-the-sky”, and one ground platform • The aerial platform is added to augment capabilities • Aerial platform offers “look-ahead” capabilities beyond obstacles • The aerial platform moves in conjunction with ground explorer • IR beacons on the ground explorer will enable aerial following • The aerial platform has multiple cameras and US sensors • Image processing is added to the system for node metadata • Image processing is conducted on the base station only

37. Phase IV Vehicle systems remain unchanged from previous phase • Full mapping capability is added to the system, no base map • Image processing can now be conducted on the aerial platform • Image processing can be conducted on the ground explorer • Navigation calculations can be performed on the aerial platform and ground explorer • Adaptive changes to roles occur dynamically during mission

38. Phase V One or more ground explorer vehicle is added to the system • Two or more ground explorers now share the mapping mission • Aerial platform can move independently of a ground explorer • Image processing, Navigational decisions, and other actions move dynamically between the base station and the explorers • Base station can move semi-autonomously for communications • Explorers act as communications relays to extend system range • RFID technology integrated into system as landmark metadata • Other sensor technologies can be added to make this a modular, plug and play system, tailored to specific mission requirements

39. Dissertation contributions

40. Dissertation Contributions • Development of Dynamically Adaptive Coordination Algorithm (efficiently manage extreme system complexity) • Development of indoor autonomous navigation techniques for airships • Exploitation of systematic advantages from heterogeneous aerial and ground system • Integration of RFID into autonomous navigation • Benefits of fixed infrastructure without limitations • Develop mechanism for deployment of tags • Develop “Perch and Stare” capability for DARPA

41. timeline

42. Projected Timeline • JUL 06 - DEC 2007: Completed required Graduate coursework • JAN 08 - JUL 08: Completed all PhD Qualifying Exams • AUG 08 - OCT 08: Completed initial research and feasibility study • OCT 2008: Present PhD Prospectus to committee • OCT 08 - DEC 08: Complete PhD Phase I & II • DEC 2008: Draft Phase II paper and submit to conference • DEC 08 - MAR 09: Complete PhD Phase III • MAR 2009: Draft Phase III paper/submit to conference • MAR 09 - MAY 09: Complete PhD Phase IV • MAY 2009: Draft Phase IV paper/submit to conference • JUN 09 - AUG 09: Complete Dissertation and Defend

43. Q & A

44. conclusion