Status as of Last Review

4. Status as of Last Review • Development • Channelizer localization • Orientation dependent colormodel • Evaluation of wide-FOV camera(product line camera)

5. Progress Since Last Review • Development • Shape filtering • Kernel-based sign tracking • Preliminary results of curb detection

6. Curbs

7. Damaged by Curbs

8. Objectives • Develop reliable methods of detecting, localizing, and classifying sufficient set of indicative features associated with curbs using in-vehicle vision sensor with backward looking view • Learn to recognize the curbs to know • Understand an appropriate parking spot as defined by the curbs when reverse or parallel parking • Identify the boundary of a road way in urban driving

9. Approach • Extend prior methods (multi-feature classification) • Use visual methods of detection (color, texture, shape) • Learn classification from large training set • Utilize camera calibration information to understand 3D geometry and change view points • Parallelize for multiple features

10. Approach • Extend prior methods (multi-feature classification) • Use visual methods of detection (color, texture, shape) • Learn classification from large training set • Utilize camera calibration information to understand 3D geometry and change view points • Parallelize for multiple features

11. Framework of Highway Workzone Recognition INPUT: A sequence of highway images Detection : Localize relevant signs in each image Tracking : Localize the detected sign in remained images before it disappears Classification : Identify the types of detected signs : Infer the current driving region based on the results of sign classification so far Inference OUTPUT: What is the road condition now? “Normal-highway” or ”Work-zone”

12. Detection Detection Tracking Tracking Classification Classification Log-polar Transform Sign Classifier Detection Classification Input Image at t Detection Classification Pixel-wise Classification Connected Component Grouping Non-maximum Suppression

13. Detection Tracking Tracking Classification • The sign image sub-regions from two consecutive image frames overlap each other. • Small variations of their appearances and locations.

14. Detection Tracking Tracking Classification Log-polar Transform Sign Classifier Detection Classification Input Image at t Pixel-wise Classification Connected Component Grouping Non-maximum Suppression Tracking Tracking Target PDF for t+1 Input Image at t+1 Sub-region Sampling Candidate PDF Candidate Localization Choose Highest Score Target PDF for t+2

15. Performance of sign detection per frame Performance of sign classification • The first row represents precision / recall of ‘detection and tracking’ • The second row represents precision /recall of ‘detection only’

16. Workzone Recognition • Created framework for detection and tracking based on color and shape • Achieved detection precision of 98.2% (with recall 88.5%) • Trained classifiers from DOT uniform signage code and from real road imagery • Achieved classification precision for signs of 96.5% (with recall 95.7%) • Demonstrated on-road, real-time recognition of work zones • Given high precision and recall, ‘Highway Workzone Recognition’ is ready for tech transfer.

17. Framework of Highway Work Zone Recognition Curb Detection Highway Workzone Recognition INPUT: A sequence of roadway images Detection : Localize relevant curbs in each image Tracking : Localize the detected curbs in remained images Classification : Identify the types of detected curbs : Infer the detected curb based on the results of classification Inference OUTPUT: Where is the curb? What does this curb mean?

18. Approach • Extend prior methods (multi-feature classification) • Use visual methods of detection (color, texture, shape) • Learn classification from large training set • Utilize camera calibration information to understand 3D geometry and change view points • Parallelize for multiple features

19. Concept • Multiple cameras (rear, side, forward) • Recognize important cues of curbs • Estimate relative position from vehicle • (Potential) Understand appropriate parking spots as defined by the curbs to assist self-parking system

20. Potential Features 3D Structure Color Segmentation Texture Classification Edge Detection

21. Structure from Motion

22. Color Segmentation

23. Edge Detection Distorted Undistorted Edge Bird’s-eye view Edge

24. Texture Classification

25. Complex Features • Ground plane estimation • To exploit 3D geometry • Edge continuity • To interpret curvy curbs • Curb scale and geometry • To utilize the height of curb above the ground plane

26. Color Classification Yellow Curb Zones Blue Curb Zones Normal Curb Zones White Curb Zones Green Curb Zones Red Curb Zones * Images from LADOT

27. Development Plan • Develop and test simple features • Train classifiers to detect and localize curbs • Evaluate classifier performance • Add complex features • Test quantify detection and localization performance • Train color classifiers to interpret appropriate parking spots

28. Questions or Comments? Acknowledgements This project is sponsored by GM-CMU AD-CRL.