1. This presentation will probably involve audience discussion, which will create action items. Use PowerPoint to keep track of these action items during your presentation • In Slide Show, click on the right mouse button • Select “Meeting Minder” • Select the “Action Items” tab • Type in action items as they come up • Click OK to dismiss this box • This will automatically create an Action Item slide at the end of your presentation with your points entered. Course Projects Topics and Plans Martin Jagersand

2. Today • Some tips on carrying out your projects • Literature search and readings • Quick prototyping (e.g. Matlab) , then final implememntation (e.g. c, c++), or combining Matlab with c, c++ mex files. • Balance between reading and doing • Labs and resources available • Win free trip? • Short presentation and discussion of your topic and plans

3. Preliminary project topics • Individual and group aspect: • Every person has some individual focus and all individual pieces combine to a whole. • Main topics: Vision for 3D modeling and robotics • Real time tracking • Integrating tracking with 3D modeling • Integration of a-priori knowledge in 3D modeling • Predictive display and visualization • Visual servoing for robots (manipulator or mobile) • Visual specification and planning of robot tasks

4. Resources • SW: I/we will install and try to help support: • Real time video input (under linux, video pipeline done) • Basic tracking, XVision • Geometry, Hand-eye, Robotics code • HW: Access to machines in Robotics/Vision lab • Cameras: Linux IEEE 1394 in both grad labs and course labs csc235. • Web cams, 200Hz high speed cams, 1600x1200 Hi-Res cam available • VZ motion tracker (3000Hz special device) • Could also use digital still camera, camcorder. • Vision for motion control: Robot arms, hands, mobile robot • WAM, Barrett hand, Segway (one dedicated more coming), old Pumas • Vision and haptics: 3 Phantom omnis. • Visualization: Ok in lab (HW acc graphics), In research lab (new ATI and nvidia, SGI HW, projectors and CAVE) • Anything else? Some resources for buying available

5. Martin’s tips • Plan incremental progress and checkpoints. • Makes it easier to identify promising directions as well as difficulties and redefine plans as needed. • Find balance between reading and doing • It is difficult to fully grasp methods by only reading • Some experiments are incomplete, results wrong • Practical trying out can add a lot of insight. • Learn how to quickly prototype in e.g. matlab

6. Literature search • Goal: Find the 10-15 most relevant and recent papers in a subarea. • Method: • Seed with a few relevant papers. • Do internet search. e.g. “research index” or google scholar • Do citation search backwards and forwards. • Find common “buzz words”. Do title and abstract text search. • Check most recent proceedings manually. (They won’t be indexed yet)

7. Literature search 2 • Expect to: • Read the titles of hundreds of papers (and web pages) • Read the abstract of 20-40 of papers • Skim through dozens of papers • In order to find the 10-15 or so relevant papers. Read these in detail to understand the topic. • Of these select a handful of the most closely related to benchmark your project to.

8. Report: • Review • Summarize the main contributions and comparing the results in the papers. • Your contribution and experiments. • Methods • Results • Discussion • Where does it fit into the bigger picture • Future work

9. Schedule • Now good time to think about and refine project plans • Late Oct Written project proposal. • Include reference list and a start at literature review, ie. Read some papers and write a few pages summary • Throughout course in class: • Keep up to date on your project progress. • In class presentation of project readings and analysis • End of semester: Project reports.

10. Preliminary project topics • Objective: Vision for 3D modeling and robotics • Individual parts • Real time tracking • Integrating tracking with 3D modeling • Integration of a-priori knowledge in 3D modeling • Predictive display and visualization • Mapping and navigation for mobile robots • Visual servoing for manipulators • Visual specification and planning of robot tasks • Singularity and obstacle avoidance • Subgroups: Vision, Robotics

11. GPU accelerated visual tracking • Tracking readings: Color, Feature, SSD, SIFT. • Investigate what maps naturally to GPU, CPU • Make incremental plan, e.g. • Video pipeline: Cam->Video RAM or CPU RAM? • Basic image processing on GPU: Lin alg, conv, filt, im deriv • Implement and test various tracking: Color, SSD, SIFT • Integrate with other system parts • Design and carry out experiments: Test sequences robustness, accuracy etc.

12. Tracking and 3D modeling • Make tracking more robust by restricting 2D image tracker movement to those consistent with a 3D interpretation • Rigid constraints: • Loose constraints • Language for partial constraints (Collab with visual spec) • Convergence tradeoff: • Restricted tracker may not reach elliptical point • Unrestricted tracker may track wrong points • Experiments: What are good test sequences? How accurate is tracking? Captured 3D? How robust?

13. Use of a-priori knowledge in modeling • Can make 3D capture easier: • Tells if recovered scene is “probable” • Orth, plan etc: and gives Euclidean structure w/o cumbersome “self calibration” • Types of a-priori knowledge: • Generic: orthogonality, parallelism, planes. • Specific: architectural: houses&features(doors windows) Indoors: Furniture on floor, lamps on wall, scales: room, furn, items… • How to mathematically incorporate: • Hard constraints • Probabilistic • How to practically add: • Image editor collaborate with Visual spec project • Experiments: • Scenes from photos, • Indoor scenes from video

14. Predictive DisplayVisual User Interface • Systems oriented project • How to modify 3D capture to incrementally detect changes in 3D remote scene, send and incorporate in model • How to display 3D model in HMD • Minimize latencies • How to track and interpret human motions. Control robot motion based on these. • Could also be set up in CAVE

15. Visual servoing of manipulators • Manipulators: Arm’s hands, high DOF devices • Investigate properties of measurement: • E-functions, properties of robot. • How to estimate visual-motor: • kinematics? Dynamics? • Local, global? • Design controllers • Combine joint and visual feedback • Integrate with tracking and visual space specifications • Experiements.

16. Visual specification • What are natural visual task primitives? • What tasks do the solve? Completeness under particular geometry? • How “smooth” E-functions do they give? (Collaboration with visual servoing). • How to have human enter: • Pointing 2D image editor • and gesturing in 3D • Experiments and test in vision-based manip

17. Singularity and Obstacle avoidance • Investigate what aspects of calibrated Euclidean approach carry over • How to find and characterize “difficult regions” • Find: Computer vision, contact, other sensing • Region representation: Points, Regions, Potential function • How to combine with visual servoing controller • Primary or secondary controller objective • Effects on convergence • Experiments with synthetic data, real sequences

18. Next steps • Firm up project ideas, specification • Identify readings • Discuss plans • Write project proposal • Investigate hardware needs and plan use • Plan interaction with other people/groups • Iterate: Devise method, implement, test • Final integration with other parts • Write final report