Decision Making and Reasoning with Uncertain Image and Sensor Data

1. Decision Making and Reasoning with Uncertain Image and Sensor Data Pramod K Varshney Kishan G Mehrotra Chilukuri K Mohan Syracuse University

2. Outline • Introduction • Main overall themes • Results since last review • Scenario Recognition: Audio Visual Sensor Fusion • Path Planning for dynamic military applications • Concluding remarks

3. Information Acquisition andFusion Model for Visualization • Dynamic network connectivity with varying bandwidths • Mobile agents with heterogeneous resources and capabilities

4. Our Main Overall Themes • Decentralized inferencing • Data/information fusion • Uncertainty representation and visualization • Planning and decision making in dynamic battlefield environments.

5. Outline • Introduction • Main overall themes • Results since last review • Scenario Recognition: Audio Visual Sensor Fusion • Path Planning for dynamic military applications • Concluding remarks

6. Objectives • Develop a proof of concept of a system, which • Classifies Activity based on • Video based detection and tracking of moving objects • Detects and classifies situational sounds • Fuses information from two different modalities to provide enhanced scene context • Handles issues such as uncertainty in sensor data and coupling between events in different streams

7. System Block Diagram Scene Descriptor Video Processing Video Fusion Framework Audio Processing Audio

8. Video Processing Pipeline Image Acquisition Background Subtraction Detection Feature Extraction Activity Classification

9. Video Processing Pipeline • Video Features • aspect ratio • speed • relative densities of pixels in upper middle and lower bands • Activity classes include walking, sitting, bending, etc • Classifier is a multi-module back-propagation neural network

10. Example: Multi Object Tracking Multi-Object Tracking in Infra Red Modality

11. Example : Tracking with ID Tags • Unique ID assigned to each tracked object. • Tracking using object properties and last known location.

12. Head Tracking for Improved Performance • Head is tracked separately • Maintain tracking of individuals in groups • Locate head in top 1/6 of the object.

13. Audio Processing Pipeline Audio Acquisition Histogram Features LPC / Cepstral Coefficients Spectral Features Relative Band Energies Choice of Features Audio Event Classification

14. Audio Processing Pipeline • Sound Classes • Silence/Background Hum • Machine noise • Alarm sounds • Human Speech • Features used for sound classification • Amplitude Histogram Features • Spectral Centroid and Zero Crossing Rate • Spectrum shape modeling coefficients • Relative Band energies • Linear Predictive Coding Coefficients • Cepstral Coefficients

15. Audio Video Fusion • Having defined the processing pipeline for the two modalities, we • develop a framework for information fusion from sensors • apply to the surveillance domain • recognize different scenarios.

16. Fusion Approach Two steps: • Decisions regarding certain activities or events are made in each information stream, based on low level data processing. • Fusion of these stream-level decisions takes place, involving three main challenges: • Asynchronism • Linkage • Uncertainty

17. Asynchronism • Events in different streams are processed asynchronously– e.g., Video events are detected on a per frame basis; and Audio events are detected over a period of time. • This asynchronism makes it challenging to fuse information from different modalities

18. Linkage • The information sensed in different streams may not be independent; it may describe the same event. • Framework for fusion must accommodate causal coupling between events across streams.

19. Modeling Linkages Correlation analysis on the training data is used to extract linkage information between features of different sensor streams.

20. Fusion Model

21. Fusion Model Video / IR sensor Sound Sensor k Inference : Theft in Progress!! Inference: running

22. Stream Model • At fixed intervals δi , decisions oi regarding the presence of events are made for the mth stream by classifiers. We use trained multi-module feed-forward neural networks to make these decisions. • At time instants k, a decision O* is calculated using the decisions (oi’s) available in that time interval for a given stream: a fuzzy rule-based approach facilitates computation of O*).

23. Fusion Framework • Fusion rules are generated using the Linkage Information learnt from the training data. • When events with stronger linkages are detected, we write the sub-scenario being corroborated by both events, whereas weakly linked events are written as separate events ex : running and alarm sound (0.1)-> possible suspicious activity bending and human speech (0.01)->2 uncorrelated events

24. Fusion Framework (continued) • Certainty values for sub-scenario observations are modified incrementally based on Linkage information. • A time series of sub-scenarios is generated, giving a complete scenario description over a period of time.

25. Illustrative Example 1 • Parking lot setting.

26. Video Sensor Input Raw video of a staged robbery Processed video of the robbery

27. Description Generated with only Video Information

28. Audio Sensor Input

29. Description Generated with only Audio Information

30. Description Generated with Audio and Video Information

31. Illustrative Example 2 Conversation

32. Visualizing other Information • Some scene variables can be visualized in sensor space • Decision Uncertainty • Threat levels • Classes of Moving Objects • Classes of Activity

33. Uncertainty Visualization • Modulate bounding box brightness to indicate object class Brightness of bounding box indicates certainty Color of bounding box indicates whether single person or group

34. Uncertainty Visualization • Bar Indicators to indicate object class Height of Bar indicator proportional to certainty Color of the bar indicates whether single person or a group

35. Summary • Demonstrated a framework for fusion of audio and video information. • Fusion of information from sensors of different modalities provides a richer scene context . • Use of probabilistic models for fusion and feature level fusion being considered. • We have shown the feasibility of activity recognition using combined video and audio information. • Next section (path planning): after activity recognition, battlefield decision-maker must act.

36. Outline • Introduction • Main overall themes • Results since last review • Scenario Recognition: Audio Visual Sensor Fusion • Path Planning for dynamic military applications • Concluding remarks

37. Path Planning in a Battlefield • To determine safe paths for personnel in a battlefield • The battlefield is represented as a graph where nodes correspond to different geographical locations with risk values • The quality of a path is measured by cumulative risk associated with the path

38. Problem Formulation • Path P : A non cyclic sequence (L1,L2….Ln) where L1 is the initial location of personnel, Ln is a target or exit point, and each Liis adjacent to Li+1 in the graph. • Determine safe paths which maximize path quality Q(P) defined as k Q(P)=  log(1-risk(Li)) i=1

39. Modeling Risks • We define risk as the probability of occurrence of a high level of damage to personnel traversing a path • Risk values at different locations can be modeled by probability distributions.

40. Optimal path computation for situational visualization (in collaboration with Bill Ribarsky) Green route: the optimal path Red semitransparent circle: the range of risks

41. Hierarchical Path Planning in Dynamic Environments

42. Problem Formulation • To compute near-optimal paths from a source to a destination with minimum computational effort. • The battlefield is modeled via a graph where the nodes represent different geographical locations • Quality measure: The cumulative risk associated with the path

43. Why Hierarchical Path Planning? • Non-hierarchical approaches such as Dijkstra’s algorithm are computationally very expensive for graphs with large number of nodes. • Hierarchical approaches • Solve the path planning problem in a hierarchcial graph with smaller number of nodes • Minimize the computational effort which is critical in real time applications such as battlefield path planning

44. Our Approach • Partition the original graph into different subgraphs and compute representative risks for each subgraph • Higher level path: compute a path in the hierarchical graph where each node represents a subgraph • Lower level path: Compute actual paths within each subgraph. The final path is a concatenation of the lower level paths in different subgraphs

45. Illustration of a Path Computed by HIPLA: Edge connecting boundary nodes Boundary node Source, destination nodes Sub path

46. Dynamic Scenario • Risk values associated with different locations change with time • Example: A segment of the path may become more risky due to the occurrence of a new event such as an explosion • Problem: compute a new path from the current location of the personnel to the destination node.

47. Our Solution for the Dynamic Path Planning Problem • Re-estimate the representative risk values only for subgraphs whose risk values have changed • Refine the current path by recomputing a new path from the current node to the destination bypassing the subgraphs whose risk values have increased.

48. An Illustration of the Dynamic Scenario Current location New path Current path Source, destination nodes

49. Simulation Results • We compared HIPLA with two other well known path planning algorithms viz., hierarchical shortest paths algorithm (SPAH)[Jung et al., IEEE trans. KDE, 2002]and Dijkstra’s algorithm with pruning (DP)[Wagner et al., Journal of Experimental algorithmics, 2005] • HIPLA obtains near-optimal solutions (worst case quality penalty within 5%) with much less computational effort compared to DP and SPAH

50. Quality Degradation of HIPLA Compared to the Optimal Solution * USA road map available at: http://www.cs.princeton.edu/courses/spring05/cos226/assignments/map/