Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot

1. Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield The Holcombe Department of Electrical and Computer Engineering Clemson University { yinxial, stb }@clemson.edu

2. Typical Corridor Images Low contrast Common Reflections Specular Reflections

3. Outline • Previous work • Detecting Line Segments • Score model for Evaluating Line Segments • Structure Score • Bottom Score • Homogenous Score • Experimental Results • Conclusion

4. Outline • Previous work • Detecting Line Segments • Score model for Evaluating Line Segments • Structure Score • Bottom Score • Homogenous Score • Experimental Results • Conclusion

5. Previous Work Free space detection Floor detection Previous approaches Apply planar homographies to optical flow vectors (Kim 2009, Zhou 2006) Stereo homographies (Fazl-Ersi 2009) Geometric reasoning (Lee 2009) Limitations Assume ceiling visible Computational efficiency Calibrated cameras Previous approaches • Stereo Matching (Sabe 2004) • Optical Flow (Stoffler 2000, Santos-Victor 1995 • Combination of color and histogram (Lorigo 1997) Limitations • Need motion • require different colors for floor and wall • Computational efficiency • Calibrated cameras

6. Outline • Previous work • Detecting Line Segments • Score model for Evaluating Line Segments • Structure Score • Bottom Score • Homogenous Score • Experimental Results • Conclusion

7. Douglas-Peucker Algorithm • An algorithm that: • Reduces the number of points in a curve that is approximated by a series of points • Focus on maximum distance between the original curve and the simplified curve • Finds a similar curve with fewer points http://en.wikipedia.org/wiki/Ramer%E2%80%93Douglas%E2%80%93Peucker_algorithm David Douglas & Thomas Peucker, "Algorithms for the reduction of the number of points required to represent a digitized line or its caricature", The Canadian Cartographer 10(2), 112–122 (1973)

8. Douglas-Peucker Algorithm • An algorithm that: • Reduces the number of points in a curve that is approximated by a series of points • Focus on maximum distance between the original curve and the simplified curve • Finds a similar curve with fewer points http://en.wikipedia.org/wiki/Ramer%E2%80%93Douglas%E2%80%93Peucker_algorithm David Douglas & Thomas Peucker, "Algorithms for the reduction of the number of points required to represent a digitized line or its caricature", The Canadian Cartographer 10(2), 112–122 (1973)

9. Douglas-Peucker Algorithm • An algorithm that: • Reduces the number of points in a curve that is approximated by a series of points • Focus on maximum distance between the original curve and the simplified curve • Finds a similar curve with fewer points http://en.wikipedia.org/wiki/Ramer%E2%80%93Douglas%E2%80%93Peucker_algorithm David Douglas & Thomas Peucker, "Algorithms for the reduction of the number of points required to represent a digitized line or its caricature", The Canadian Cartographer 10(2), 112–122 (1973)

10. Douglas-Peucker Algorithm • An algorithm that: • Reduces the number of points in a curve that is approximated by a series of points • Focus on maximum distance between the original curve and the simplified curve • Finds a similar curve with fewer points http://en.wikipedia.org/wiki/Ramer%E2%80%93Douglas%E2%80%93Peucker_algorithm David Douglas & Thomas Peucker, "Algorithms for the reduction of the number of points required to represent a digitized line or its caricature", The Canadian Cartographer 10(2), 112–122 (1973)

11. Douglas-Peucker Algorithm • An algorithm that: • Reduces the number of points in a curve that is approximated by a series of points • Focus on maximum distance between the original curve and the simplified curve • Finds a similar curve with fewer points http://en.wikipedia.org/wiki/Ramer%E2%80%93Douglas%E2%80%93Peucker_algorithm David Douglas & Thomas Peucker, "Algorithms for the reduction of the number of points required to represent a digitized line or its caricature", The Canadian Cartographer 10(2), 112–122 (1973)

12. Douglas-Peucker Algorithm • An algorithm that: • Reduces the number of points in a curve that is approximated by a series of points • Focus on maximum distance between the original curve and the simplified curve • Finds a similar curve with fewer points http://en.wikipedia.org/wiki/Ramer%E2%80%93Douglas%E2%80%93Peucker_algorithm David Douglas & Thomas Peucker, "Algorithms for the reduction of the number of points required to represent a digitized line or its caricature", The Canadian Cartographer 10(2), 112–122 (1973)

13. dallowed Modified Douglas-Peucker Algorithm original algorithm: dallowed is constantmodified algorithm: dallowed is given by half-sigmoid function original modified

14. Detecting Line Segments (LS)

15. Detecting Line Segments (LS) Compute Canny edges

16. dallowed Detecting Line Segments (LS) • Compute Canny edges • Modified Douglas-Peucker algorithm to detect line segments • Vertical LS: slope within ±5° of vertical direction • Horizontal LS: Slope within ±45° of horizontal direction

17. dallowed Detecting Line Segments (LS) • Compute Canny edges • Modified Douglas-Peucker algorithm to detect line segments • Vertical LS: slope within ±5° of vertical direction • Horizontal LS: Slope within ±45° of horizontal direction • 3. Pruning line segments (320x240) • Vertical LS: length 60 pixels • Horizontal LS: length 15 pixels • Vanishing point (height selection):The vanishing point is computed as the mean of the intersection of pairs of non-vertical lines

18. dallowed Detecting Line Segments (LS) • Compute Canny edges • Modified Douglas-Peucker algorithm to detect line segments • Vertical LS: slope within ±5° of vertical direction • Horizontal LS: Slope within ±45° of horizontal direction • 3. Pruning line segments (320x240) • Vertical LS: length 60 pixels • Horizontal LS: length 15 pixels • Vanishing point (height selection):The vanishing point is computed as the mean of the intersection of pairs of non-vertical lines

19. Outline • Previous work • Detecting Line Segments • Score model for Evaluating Line Segments • Structure Score • Bottom Score • Homogenous Score • Experimental Results • Conclusion

20. Score Model Weights • is assigned to each horizontal line segment and we use a threshold to select those ones that beyond the Structure Score Homogeneous Score Bottom Score

21. Score Model – Structure Structure =>Threshold=? Given a typical corridor image

22. Score Model – Structure Structure =>Threshold=? Given a typical corridor image Find gradient magnitude greater than a threshold T1

23. Score Model – Structure Structure =>Threshold=? • Given a typical corridor image • Find gradient magnitude greater than a threshold T1 • Find gradient magnitude greater than a threshold T1 and intensity less than a threshold T2. The figure shows the mask pixels image. • We have a training data over 200 images to determine T1 and T2 by SVM over 800 points.

24. Score Model – Structure Structure =>Threshold= TLC Given a typical corridor image Find gradient magnitude greater than a threshold T1 Find gradient magnitude greater than a threshold T1 and intensity less than a threshold T2. The figure shows the mask pixels image. Compute the average intensity of the make pixels image over original image and get a threshold value TLC

25. Score Model – Structure Structure =>Threshold= TLC Given a typical corridor image Find gradient magnitude greater than a threshold T1 Find gradient magnitude greater than a threshold T1 and intensity less than a threshold T2. The figure shows the mask pixels image Compute the average intensity of the make pixels image over original image and get a threshold value TLC Compute the distance between the line segments and structure blocks

26. Score Model – Structure Comparison of different threshold methods Ridler-Calvard Otsu Our method is able to get rid of the spurious pixels on the floor caused by reflections or shadows Our method

27. Outline • Previous work • Detecting Line Segments • Score model for Evaluating Line Segments • Structure Score • Bottom Score • Homogenous Score • Experimental Results • Conclusion

28. Score Model – Bottom Using vertical and horizontal line segments Connect each bottom point of the vertical line segments as a “bottom” wall-floor boundary

29. Score Model – Bottom Using vertical and horizontal line segments Connect each bottom point of the vertical line segments as a “bottom” wall-floor boundary Compute the distance of each horizontal lines segments to the “bottom” wall-floor boundary (red circled horizontal line segments indicates a positive contribution)

30. Outline • Previous work • Detecting Line Segments • Score model for Evaluating Line Segments • Structure Score • Bottom Score • Homogenous Score • Experimental Results • Conclusion

31. Score Model – Homogeneous Using efficient graph-based segmentation ( Felzenszwalb [5]), we may get a bigger segmented area in floor area, due to decorations, posters, windows etc. on the wall. Compute the ratio of area of the segmented region below a horizontal line segment over the biggest segmented region in the image. The ratio is assigned to each horizontal line segment as a homogenous score.

32. Detecting wall-floor boundary Given a threshold to the sum of each score Connect the remaining line segments and extend the endpoints to the edges of the image Future work: A larger database and AdaBoost algorithm to generate weights for each score

33. Outline • Previous work • Detecting Line Segments • Score model for Evaluating Line Segments • Structure Score • Bottom Score • Homogenous Score • Experimental Results • Conclusion

34. Evaluation Criterion • Wall- floor Boundary • Green line: Detected wall-floor boundary • Red line: Ground truth

35. Evaluation Criterion • Wall- floor Boundary • Green line: Detected wall-floor boundary • Red line: Ground truth • Area • Blue shade area: misclassified area • Red shade area: ground truth floor area • Error Rate

36. Sample Results Corridor images from our database

37. Sample Results Images download from internet.

38. Sample Results Lee’s Original Image Ours Some successful results of failure examples from Lee[1].

39. Sample Results Texture wall Checked floor Overly dark Bright lights Failure examples

40. Sample Results Video clip 1

41. Sample Results Video clip 2

42. Outline • Previous work • Detecting Line Segments • Score model for Evaluating Line Segments • Structure Score • Bottom Score • Homogenous Score • Experimental Results • Conclusion

43. Conclusion • Summaries • Over 90% of the corridor images in our database can be correctly detected. • Speed: around 7 frame/sec. • Weakness: not robust to highly textured floor, low resolution and dark environment. • Future work • Speed up the current algorithm • Improving score model by add more visual cues • Mapping the whole corridor structure for robot navigation • …

44. Acknowledgement • Clemson Computer Vision Group reviewers • Zhichao Chen • Vidya Murali • Anonymous reviewers