Multisensor Data Fusion and Applications

1. Multisensor Data Fusion and Applications

2. Outline • Overview of Multisensor Data Fusion • Distributed Detection and Decision Fusion • Applications • Integrated Vehicle Health Management • Monitoring and Control of Built Environments • Image Fusion • Some Current Research Activities • Concluding Remarks

3. Information Fusion Theory, techniques, and tools for exploiting the synergy in the information acquired from multiple sources: sensors, databases, intelligence sources, humans, etc.

4. Information Fusion Goals: • To obtain a better understanding of some phenomenon • To introduce or enhance intelligence and system control functions

5. Taste Sight Smell Hearing Touch Human Brain Integrates sensory information to make inferences regarding the surrounding environment.

6. Advantages of Multisensor Data Fusion • Improved system performance • Improved detection, tracking, and identification • Improved situation assessment and awareness • Improved robustness • Sensor redundancy • Graceful degradation • Extended spatial and temporal coverage • Shorter response time • Reduced communication and computing

7. Applications - Military • Detection, location , tracking and identification of military entities. • Sensors: radar, sonar, infrared, synthetic aperture radar (SAR), electro-optic imaging sensors etc. • Complex problem • Large number and types of sensors and targets • Size of the surveillance volume • Real-time operational requirements • Signal propagation difficulties

8. Applications - Non-military • Air traffic control • Law enforcement and homeland security • Medical diagnosis • Smart homes, buildings and bridges • Critical infrastructures • Highways • Water distribution networks • Robotics • Manufacturing • Hazardous workplace • Remote sensing • Crops • Weather patterns • Environment • Mineral resources • Buried hazardous waste

9. Key Issues for Fusion Systems • Nature of sensors and information sources • Location - co-located or geographically distributed • Computational ability at the sensors • System architecture - topology, communication structure, computational resources, fusion level • System goals and optimization

10. Key Issues for Fusion Systems • Data level fusion • commensurate sensors, centralized processing • Examples: image fusion, tracking • Feature level fusion • feature extraction, reduced communication bandwidth or computing requirement • Examples: biometrics, remote sensing • Decision level fusion • performance loss, easy to fuse heterogeneous information sources • Examples: distributed detection, classifier fusion, tracking

11. Key Issues for Fusion Systems

12. Fusion Techniques for Multisensor Inferencing Tasks Techniques • Existence of an entity • Identity, attributes and location of an entity • Behavior and relationships of entities • Situation Assessment • Performance evaluation and resource allocation • Signal detection/estimation theory • Estimation and filtering, Kalman filters • Neural networks, Clustering, Fuzzy logic • Knowledge-based systems • Control and optimization algorithms Fusion levels Solution of complex fusion problems requires a multi-disciplinary approach involving integration of diverse algorithms and techniques

13. 2. Distributed Detection and Decision Fusion

14. Probabilistic transition mechanism Observation space Source Decision rule Decision The Signal Detection Problem Binary hypothesis testing: determination of the presence or absence of a target (H1 vs. H0) Components of a hypothesis testing problem

15. The Distributed Detection Problem • Solution of a detection problem by a team of interconnected detectors/agents • Network architecture • Sensor placement • Fixed vs. mobile sensors • Infrastructured vs. ad hoc networks • Topology • Communication and channel bandwidth issues

16. The Distributed Detection Problem • Optimization criterion • Design of optimal signal processing schemes at various detectors and the fusion center • NP-Hard problem • Performance of centralized detection versus distributed detection

17. Phenomenon y2 y3 y1 yN DM 1 DM 2 DM 3 DM N u1 u2 u3 uN Fusion Center u0 Parallel Fusion Network ...

18. u1 Data fusion center u2 u0 ... uN Fusion Center

19. 0, if detector i decides H0 0, if H0 is decided ui= 1, if detector i decides H1 1, otherwise u0 = Design of Fusion Rules Input to the fusion center: ui, i=1, …, N Output of the fusion center: u0 Fusion rule: logical function with N binary inputs and one binary output Number of fusion rules: 22N

20. Input Output u 0 u u f f f f f f f f f f f f f f f f 1 2 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 0 1 0 0 0 0 1 1 1 1 0 0 0 0 1 1 1 1 1 0 0 0 1 1 0 0 1 1 0 0 1 1 0 0 1 1 1 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 Possible Fusion Rules for Two Binary Decisions

21. The optimum fusion rule that minimizes the probability of error is

22. For more information, please refer to Pramod K. Varshney, Distributed Detection and Data Fusion, Springer-Verlag New York, Inc., 1997.

23. Decision Level Fusion: Biometric Application Each sensor decides to accept or reject the individual prior to fusion . A global decision is made at the fusion center.

24. Performance Improvement Due To Fusion Fusion of two face-matchers: from S. G. Iyengar, P. K. Varshney and T. Damarla, “Biometric Authentication: A Copula Based Approach,” in Multibiometrics for Human Identification (B. Bhanu and V. Govindraju Eds.), Cambridge University Press (2012).

25. 3. Applications • Integrated Vehicle Health Management • Monitoring and Control of Built Environments • Image Fusion

26. Data Fusion for Integrated Vehicle Health Management (IVHM) • Importance of IVHM • Prevent catastrophic failure • Reduce the cost and downtime associated with maintenance • Desired Functions of IVHM • Monitoring • Anomaly detection • Diagnosis • Prognosis • Fault tolerance and dynamic reconfigurability

27. Distributed Systems for IVHM • Advanced future aircrafts will be large-scale distributed systems with 109 to 1023 of interconnected components • Smart sensors • Processing nodes • Smart actuators • Data fusion centers • Develop and manage complex large-scale cooperative distributed systems • Complex Topologies • Dynamic Reconfigurability • Ultra-High Reliability • Dependable embedded real-time monitoring, error detection and recovery, control and diagnostic Functions

28. Distributed Fault Detection

29. Distributed Fault Detection • Model based fault detection at sensors • Change/No change decisions transmitted to the fusion center for decision fusion • Several examples show the enhanced performance due to data fusion including a B737 aircraft system • See Cheng, Varshney et al. in IEEE T-AES Jan 2008

30. 3. Applications • Integrated Vehicle Health Management • Monitoring and Control of Built Environments • Image Fusion

31. PEOPLE AND FURNITURE • Paint, carpet emit VOCs • Clothes/Grooming Products • SMOKING • Circulates through the ventilation system • EXTERMINATORS • Pesticides contain carcinogens • WHAT FRESH AIR? • Vents located over loading docks • SEALED WINDOWS • No access to outdoor air • CARCINOGENIC PRODUCTS • 70,000 chemical cleaning products on the market • COPY MACHINE AND PRINTERS • Emit Ozone • THE OFFICE BATHROOM • Mold machine • BUILDING RENOVATIONS • Paint fumes, dust, odors Do you work in a Toxin Factory?* *Business Week June 5, 2000 Indoor Air Pollution

32. Introduction and Motivation • Health • 17.7 million asthma cases (4.8 million children) • 50-100 thousand annual deaths due to elevated levels of particulate matter • Productivity • $40 to $250 billion productivity loss due to poor indoor environmental quality (IEQ) • Sustainability • $110 billion annual economic loss due to air pollution in urban areas • 40% of total building energy consumption is for environmental control (over 15% of total US energy consumption) • Security • Built and urban environments are vulnerable to chemical/biological threats

33. ICUBE picture

34. Wireless Sensor Unit Front, Side and Inside views of our WSN units

35. 3. Applications • Integrated Vehicle Health Management • Monitoring and Control of Built Environments • Image Fusion

36. Fusion Extraction Recognition Interpretation Multisensor Image Processing Steps INPUT Preprocessing  Noise Removal Registration OUTPUT

37. Example 1 Multisensor Image Fusion Visual IR Fused

38. Example 2 Hyperspectral Image Fusion Band 1 Band 61 Band 121 Fused 1

39. IR Fused Segmented MMW Example 3: Concealed Weapon Detection Application

40. Example 4 Medical Image Fusion MR Fused CT

41. Example 5 Multi-focus Image Fusion Fused

42. 4. Some Current Research Activities

43. Wireless Sensor Networks • WSNs integrate a large number of low cost computationally-limited processors. These processors have flexible interfaces allowing various sensors to be networked. Fusion Center Sensor and Local processor Ad Hoc Network Topology

44. HEALTH MONTIORING (BODY SENSOR NETWORKS) AVIATION DIAGNOSTICS DEFENSE (DISTRIBUTED RADAR, UGS) ECOLOGICAL MONITORING

45. Six Blind Men and an Elephant It was six men of IndostanTo learning much inclined,Who went to see the Elephant(Though all of them were blind),That each by observationMight satisfy his mind.The First approached the Elephant,And happening to fallAgainst his broad and sturdy side,At once began to bawl:"God bless me! but the ElephantIs very like a wall!"…… And so these men of IndostanDisputed loud and long,Each in his own opinionExceeding stiff and strong,Though each was partly in the right,And all were in the wrong!

46. Research and Development Issues • Networking Issues • Initialization, architecture, protocols, placement, fault-tolerance, security • Signal Processing Issues • Detection, estimation, tracking, recognition, compression, fusion, collaboration • System Constraints • Bandwidth, energy, time • Sensor and Resource Management

47. A Copula Framework for Multisensor Fusion (ARO) • Typical assumptions in the Fusion literature • Homogeneous or commensurate sensing modalities • Statistical independence between sensors’ data • We relax both assumptions using the statistical theory of copulas • Copula theory enables us to • Characterize the joint statistics of possibly disparate marginal distributions (e.g., audio and video data) • Obtain more general descriptors of dependence

48. Detection of Personnel Activity Detect personnel activity using non-stationary, heterogeneous, dependent sensor data • Environment • Video surveillance is not possible • Wind, stray ground vibrations: noisy background • Design • Sensors are not of the same type: heterogeneous data. How to get a valid joint model? • Non-stationary signals: personnel & animal activity • Data collected at US southwest border

49. Quantification of Neural Synchrony • Neural synchrony: co-movement of neural activity • Why do we care? • Suggestive of neurophysiological disorders such as Alzheimer’s Disease and epileptic seizures • Useful for studying brain connectivity and neural coding • How do we quantify synchrony? • Limitations of existing measures • Existing measures such as Granger causality measure only the linear relationship • Information theoretic measures such as mutual information are constrained to be bivariate • Copula based multi-information rates developed to alleviate both limitations

50. Concluding Remarks • How should we identify the “elephant” using “relatively blind (as well as relatively deaf and mute)” sensors/sources? • Fusion Challenges • Homogeneous vs. heterogeneous • Single modality data vs. multi-modal data • Synchronous vs. asynchronous • Extraction of actionable information in a timely manner and delivery to humans in an easy to understand format • Fusion of “soft” data with “hard” data