Contents

1. PixelGT: A new Ground Truth specification for video surveillanceDr. Kenneth Dawson-Howe,Graphics, Vision and Visualisation Group (GV2),School of Computer Science & Statistics, Trinity College Dublin

2. Contents Introduction Review Specification Annotation Conclusions • Introduction • Review • Specification • Annotation • Conclusions

3. Available video sequences Introduction Review Specification Annotation Conclusions • Video sequences with no ground truth • Video sequences with moving object pixel masks • Video sequences with moving object bounding boxes • Video sequences with labeled events

4. Available video sequences Introduction Review Specification Annotation Conclusions

5. Video sequences with moving object pixel masks Introduction Review Specification Annotation Conclusions • Video sequences with moving object and moving shadow pixel masks. • Artificial video sequences with moving object pixel masks. • Video sequences with labeled pixels. ATON dataset PETS/LIMU dataset VSSN dataset CamVid dataset

6. Video sequences with moving object bounding boxes Introduction Review Specification Annotation Conclusions CAVIAR dataset ETISEO dataset

7. Video sequences with labeled events Introduction Review Specification Annotation Conclusions PETS 2006 dataset

8. Motivation Introduction Review Specification Annotation Conclusions • Existing ground truth requires a choice: • Binary pixel masks (with no specific object identification) OR • Object identification (with bounding boxes only and no shadows). • Relatively little pixel accurate ground truth is available. • No existing means of dealing with transparency and reflections.

9. Ground truth Specifcation Introduction Review Specification Annotation Conclusions • Pixel based ground truth • Bit organisation • Meta data • Gradient profiles • Bounding boxes & events

10. Pixel based ground truth Introduction Review Specification Annotation Conclusions • Object class (8 bits) • Object instance (9 bits) • Shadow level (4 bits) • Transparency level (3 bits)

11. Bit organisation Introduction Review Specification Annotation Conclusions L = Label class I = ID S = Shadow T = Transparency 1 = LSB 8 = MSB

12. Meta data Introduction Review Specification Annotation Conclusions • Based on CVML • Includes: • File names • Bit organisation • Meaning of object labels • Gradient profile

13. Gradient profiles Introduction Review Specification Annotation Conclusions • Width of edges? • Required for assessing performance?

14. Bounding boxes and events Introduction Review Specification Annotation Conclusions • Bounding boxes for moving objects • Implicitedly included • Can be explicitly specified • Aids comparison of techniques • Events • Some implicitedly included • Can be explicitly specified • Aids comparison of techniques

15. Creation of ground truth Introduction Review Specification Annotation Conclusions • Difficult to label all pixels in all frames • Short videos? • Not all frames / pixels? • Method • Background annotation • Object and shadow boundary specification • Object class and ID annotation • Transparency, Meta-data, Bounding boxes, Events and Gradient Profile specification • Propagation of ground truth

16. Specifying object class & ID Introduction Review Specification Annotation Conclusions

17. Background annotation Introduction Review Specification Annotation Conclusions • For static cameras • Annotate all background pixels • Facilitates automatic event detection

18. Specifying object and shadow boundaries Introduction Review Specification Annotation Conclusions • Identify boundary edges of objects / shadows • Once shadow boundary is complete shadow level is computed automatically.

19. Specifying object class & ID Introduction Review Specification Annotation Conclusions • Once boundary is complete • Select object class and (optionally) ID • Click on the object

20. Transparency, Meta-data, Bounding boxes, Events and Gradient Profile specification Introduction Review Specification Annotation Conclusions • Transparency not yet addressed • Meta data – specified in CVML • Bounding boxes & events – extract automatically from pixel based ground truth • Gradient profile – extract automatically from each image

21. Propagation of ground truth Introduction Review Specification Annotation Conclusions • Propagation only required for moving objects • Working with every frame results in very little movement between frames (0-5 pixels) • Search for the closest similar contour in the new frame where the pixels on the object side of the contour appear similar… • Need to identify & correct • when new objects enter the scene. • when objects reappear from behind occlusions. • when internal holes appear within an object. • when errors occur in the propagation

22. The positive spin… Introduction Review Specification Annotation Conclusions • PixelGT is a new form of ground truth which will facilitate • Pixel-level assessment of trackers • Semi-automatic generation of object location, bounding boxes and event descriptions. • Comparison of trackers which assess performance using different types of ground truth

23. Questions Introduction Review Specification Annotation Conclusions • Do we really need this detailed ground truth ? • Should we change anything about the specification ? • Other questions ?