Video Event Recognition Algorithm Assessment Evaluation Workshop VERAAE

1. Advanced Research and Development Activity Video Event Recognition Algorithm Assessment Evaluation Workshop VERAAE ETISEO – NICE, May 10-11 2005 Dr. Sadiye Guler Sadiye Guler - Northrop Grumman IT/TASC Mubarak Shah, Niels da Vitoria Lobo - University of Central Florida Rama Chellappa, Dave Doermann - University of Maryland US Government Champions: Terrence Adams-NSA, John Garofolo, Rachel Bowers-NIST

2. Problem • Comparative study of Video Event Recognition (VER) algorithms to assess applicability, usefulness and limitations of different approaches • Motivation: • Several promising VER algorithms exist • The algorithms have varying degrees of success with different types of event detection • No largely accepted criteria or data set (with ground truth) exist for VER evaluation (few emerging studies..) • The performance of VER algorithms is highly dependent on the results of object detection and tracking, rendering fair comparison of just the “event recognition” very difficult

3. Workshop Goals • Produce realistic operational video event data set representing scenarios for surveillance domain • Ground truth the video event data for VER and map to suitable Event Ontology developed in previous workshops • Annotatethe data set with object detection and tracking metadata that serves the needs of all participating/expected event recognition algorithms • Develop evaluation criteria and metrics for quantitative evaluation of VER algorithms and software tools for evaluation • Assess different VER approaches for the applicability to operational scenarios by their learning/explanation/ recognition capabilities

4. Event IC Event Video Video Scenarios Scenarios Data Data Annotated Annotated object metadata object metadata Event Event Video Video Ontology Ontology event metadata event metadata Data Data Video Event Video Event Evaluation Evaluation Technology Technology Recognition Recognition methodology methodology Assessment Assessment Algorithms Algorithms VERAAE Approach

5. VERAAE Object features, tracks Behaviors, actions, events Video Event Recognition and VERAAE Video data Signal Content Extraction Provided Raw Information Event Detection Evaluation Semantics, Ontology Event Recognition Abnormal and suspicious events Trends, correlations.. Knowledge, Intelligence

6. Workshop event scenarios Data Set VERAAE Domain VERAAE domain focus: surveillance realistic scenarios of interest Events and activities existing algorithms can detect Realistic high level or complex events end-users want to detect

7. Data Set Planning • Primary factors that determine the data requirements: • Fixed camera views, no PTZ • Color, B&W and IR • Realistic operational scenarios • About 10 events with varying complexity, at least 10 samples per event • The collection parameters that address the functional capabilities of the algorithms • Annotation will include the object track data required by the participating algorithms (automatically and manually generated) e.g.: • Silhouettes of tracked objects • Bounding boxes and centroid of objects (U Maryland ViPER tool) • Object category e.g. vehicle, person, box, animal,… • Ground truth for video events will be generated using the event ontology work • Frame numbers (time offsets) for Event Start and End, identified simple sub events

8. Event Ontology (Event Taxonomy workshop) • Simple event Domain independent action descriptors e.g. abandoning an object • Compound (complex or multi-threaded) event Multiple simple events taking place in time and space constraints to achieve complex activities. e.g. planting suspicious object, (if considered with below simple events moving in the wrong direction parked car at the curb-side no one exiting parked car getting in the car • Domain specific high level event • Semantic interpretation of events in a particular context, over multiple-views and multiple data type events e.g. sabotaging public facility

9. Recognizing Surveillance Events Surveillance Event types from the user’s point of view • Violation of some rule • wrong direction (in thru the out door) • abandoned object ( suitcase left unattended for t>T) • Suspicious or Interesting activity • non exit from a parked car • repeated visits to a store shelf • Abnormal activity • approaching several cars in the lot • several somewhat suspicious events in close proximity Naturally represented by rules and constraints Users can easily describe them Highly context dependent, even context from other camera views Users can not easily describe but know when they see it Naturally represented by probabilistic models and learning Users build a sense of “normalcy”

10. Recognizing Surveillance Events • Knowing what can be detected we describe the events using not only observable, but also detectable actions • Example: Shoplifting • Camera 2 in the parking lot • Car in front of emergency exit • No one exits from car • Camera 1 in the store • Repeated visit to an area • Running in the store

11. Rule Based Event:Violation by an activity constraint – car parked in the driveway

12. Rule Based Event:Violation by an object class constraint

13. Suspicious Event:“testing” the exclusion zone

14. Abnormal Event: Vehicle casing the building

15. Abnormal Event: Large Vehicle at the Gate

16. Workshop Timeline May 05 December 05 Data, Evaluation criteria generation, distribution Planning, invitations communications Evaluation tools development, Evaluation results, Final report Evaluation Criteria Focus Meeting Scenario Focus Meeting First Workshop Meeting June 20/21 With CVPR Workshop Dry-Run Meeting October (3rd week) In Boston Workshop Final Meeting Final report This is a “seedling” workshop to investigate feasibility

17. Workshop Approach • First Workshop Meeting (2 days, June): • Purpose: • Workshop goals and vision; • Presentation and determination of algorithms to participate in the workshop; • Presentation of example data sequences. • Outcome: • Outline of the data requirements (object tracking, data exchange protocols etc.) • Draft a rough set of evaluation criteria • Solicit feedback on scenario complexity and realism

18. Workshop Approach • Evaluation Criteria Focus Meeting (2 days, July): • Purpose:to determine evaluation criteria best suited for VER. • Outcome:- Evaluation Criteria will be interactively developed in workshop meetings leveraging Event Ontology, VEML and ETISEO workshop findings • Evaluation metrics at the component and system level will be defined based on • Recognition rate • Learning rate • Recall and Precision rates • True/False positives, True/False negatives and relevance of false detections • Event decomposition (based on the ontology defined sub event recognition rate)

19. Workshop Approach • Workshop Dry-Run Meeting (2 days, October 05) • Purpose and Outcomes: • Participant’s feedback on processing the sample data sets. • Evaluation tools and methodology presentation • Evaluating the “evaluation criteria” and finalizing all metrics to be used. • Planning of evaluation format • Discussion of interpretation of results

20. Workshop Results • Raw and annotated (with object detection and tracking data) video sequences for realistic operational scenarios • Event Recognition ground truth data based on surveillance Event Ontology • Re-usable and extendible Evaluation Criteria suitable for VER • Software tools for event detection evaluation • The groundwork for a formal VER evaluation process