New Folder Technology Applications and Success Stories

1. New Folder Technology Applications and Success Stories New Folder Consulting

2. Outline • Image and video processing, scene understanding • Change detection in vehicle-mounted video (Video aligning, change detection) • Automated driving (Video processing, local features, tracking, inference) • Infrared Tripwire detection (Image processing) • Pedestrian identification and tracking (Adaptive Gaussian mixture models) • Time and frequency domain pattern recognition, machine learning • Gun shot identification (New models for time-series data classification) • Chemical weapon identification (Context-dependent classification)

3. Change Detection in Video (with NVESD) What’s interesting about this image?

4. Change Detection In Video: Goal • To leverage pre-collected, vehicle mounted data to detect changes between multiple excursions over a particular road or route Day 1 Day 2 Now, what’s interesting about 2nd image?

5. Solution • Apply: • Local image features • Statistical pattern recognition • Robust statistics • Dynamic time warping • Learn optimal warping between two paths, combine and isolate differences based on local feature distributions

6. Video Example

7. Automated Vehicle Processing and Interaction (with CISCO) • Goal: • Investigate techniques for video understanding(un-solved problem) • Application: • “Play a (racing) video game competitively against a 16 year old” • Real goal is to advance video understanding, video search, and video processing for many different goals

8. Webcam Setup • Lab setup includes PS3, webcam, connected to PC • Preliminary real-time processing framework in place • OpenCV • Short-term goals: • Track player’s vehicle • Isolate mini-map • Extract information from odometer/speedometer

9. Video Game Demonstration

10. Tripwire Identification in IR Data • Tripwires provide a constant source of danger to overseas troops • Some tripwires can be found in IR imagery, but requires careful visualization • Can we provide soldiers with real-time automatic identification of tripwires in IR data? • Significant computational restrictions • Requires efficient algorithm development

11. ApproximateGround Mask Fused Black Hot 40 Channel 3 ApplyGround Mask Channel 1 Local Demeaning Gradient Based Transform Radon Transform Merge proximate lines and limit minimum length

12. Pedestrian Detection and Tracking • Identification and tracking of pedestrians has many applications in • Traffic • Security (at many levels) • Military operations • Adaptive Gaussian mixture modeling and Kalman tracking of low-likelihood regions can provide significant tracking capabilities with low computational complexity

13. Gunshot Detection (with ARO) • Goal: • Accurately identify make and model of different guns based on acoustic muzzle blast • Applications: • Military and security applications for identifying snipers • Secondary research involves accurate localization in complicated environments

14. Research • Would like to build HMM with Auto-regressive state densities to model changing time-frequency characteristics of gunshots • But what complexity model? (How many states? Weights?) • Require arbitrary complexity model for time series, but can’t search infinite space • Bayesian processing provides automatic tradeoff between model complexity and accuracy • New time-series models: • Uncertain order AR, Stick-Breaking HMM • Tractable inference • Powerful modeling tool

15. UOAR SBHMM - Acoustic Signal Modeling • Muzzle blast from a Glock Model 17 • Model inferred from 10 examples • Maximum number of states S=25 • 6 states utilized • Most AR components have a low order • (2 or 4) • Broad spectral range • UOAR SBHMM characterizes time-frequency information similar to the STFT

16. Context-Dependent Chemical Weapon Identification (With ARO & AFRL) • Goal: • Detect trace amounts of chemical weapons in vapor or solid form in varying gasses or on varying substrate • Application: • Early warning system for military and humanitarian applications • Uses LIBS based sensing • Elemental analysis tool, requiring no sample preparation • Provides spectral lines indicating elemental composition of sample under consideration • Extremely rapid analysis • Issues: • LIBS is sensitive to background matrix (context) • Context-dependent classification

17. Context Dependent Learning Incoming Data Points - x • Instead of trying to mitigate context; exploit context • Same features, information might not be ideal under different sensing scenarios • Infer context, within context, learn decision boundaries • Testing time, aggregate across all contexts weighted by likelihood given data P(C1|x) P(C2|x) P(C3|x) P(Hj|x,C1) P(Hj|x,C2) P(Hj|x,C3) Requires estimates of p(H1|x,Ci) and p(Ci|x) New Folder Consulting

18. Context Dependent Classification if Chemical Weapon Simulants in Varying Backgrounds • Context-dependent classification improves percent correct classification from 73 to 84 percent correct with relatively straightforward modifications • Improved performance, lower computational complexity, less risk of over training New Folder Consulting

19. Conclusions • New Folder has many years of experience performing statistical analysis and algorithm development in a wide range of applications • Video, GPR, Cochlear implants, Brain computer interface, LIBS, Acoustics, electromagnetic induction, seismic, etc. • Our focus is on making things work • Not equations on a piece of paper, but real solutions for real problems • How can we help you?

20. Backup Slides

21. Change Detection

22. Video Data & System • Video camera mounted on Nissan Pathfinder • Data collected in heavily wooded area near Duke campus • Objects emplaced along a 100 meter long route • Landmine simulants, UXO objects, crates, boxes, roadside explosive threat simulants etc. • Varying number of objects in each data collection

23. Local Image Features (Video) • Local image features provide concise set of information describing an image • Fast to calculate, sparse representation • Triangles and squares on right indicate locations of Harris [Harris, 1988] features from two passes • Features consistent, except where new object is encountered (ellipse)

24. Distance Metric • Need to define distance between two frames of GPR or video data • Given two sets of feature locations, {xi,yi}, from two frames (H1 and H2) in the two video streams, can define • H1 = {xi1,yi1} • H2 = {xj2,yj2} • But D(H1,H2) as defined above is sensitive to outlying feature locations ( ), which will result in large distances

25. Robust Distance Metric • Robust distance metric should not be adversely affected by certain % of outliers • Remaining work uses a one-sided trimmed mean • Reject top 30% of large distance outliers

26. Aligning Video Streams • Once feature sets extracted, need a distance between feature sets from two different frames • Many possibilities available • Need invariance under spurious features (noise features) • May require invariance under spatial shifts • Sensor height • Sensor cross-track position? • Once feature distance defined, can use DTW or similar (real-time) approaches to warping video

27. Detecting Changes (1) • Given aligned video stream, still need to identify changes • 1st match feature locations from original image to new image • Can we find regions with a high density of unmatched feature locations? ?

28. Detecting Changes (2) • Can estimate density of unmatched local features using kernel smoothing • Resulting disk-based estimate of density shown on bottom • Can use thresholded density as change-detector

29. Recall Goal • To leverage pre-collected, vehicle mounted data to detect changes between multiple excursions over a particular road or route (Video, and GPR) Day 1 Day 2

30. Change Detection in GPR • Given: 2 GPR Images at approximately the same spatial location, with alarms • Any significant change in data between passes? • Identify new targets; reject false alarms

31. Change Detection Examples: GPR • Images on right illustrate capacity of algorithm to identify changes in GPR data • GPR data streams were aligned using distance metric outlined previously • Changes identified by concentration of un-matched feature locations

32. Direct Detection Rejections • Direct detection algorithms are a large part of GPR processing • Can we “correct” direct detections that occur over objects which have not changed?

33. Change Detection in Video: 1 Frame

34. Driving

35. Video Processing: Vehicle Tracking • Vehicle tracking as a subset of object tracking • Has direct relevance to video search: • Car making right turn with clouds in distance • Car passing police car on left • Current naïve tracking uses localized matched filters with adaptive templates

36. Extracting Game Meta Data: Map Segmentation • Transparent map on left provides information about road, other vehicles • May be easier to utilize than full video data • Less information in total, but useful meta-data • E.g., can see other vehicles on map prior to on road • Our play-testing found this map very useful

37. Extracting Game Meta Data: Speedometer • Inferring speed can be difficult at high velocities • Vehicle odometer has significant information, need to perform classification of digits to infer velocity • Can also infer RPM, gear, turbo, and vehicle health

38. Supervised Learning & Digit Identification • Extracted labeled examples of digits from NFS2 video streams • Project into PCA space • Learn classifier using Gaussian probability density functions • Output MAP classification decision

39. Gunshot Detection, Weapon ID & Acoustic Signal Classification Gunshot detection and weapon ID require • Discrimination between weapons and other potential false alarms • Weapon Identification Two primary goals • Algorithm adaptability • Recursive Bayesian inference specifically with Variational Bayes • Independent of specific signals • Nonparametric Bayesian methods

40. Acoustic Signal Classification - Typical Approach • Similar to acoustic surveillance approach • Frame data (20ms – 1s) • Extract Features • Energy and spectral features • Perceptual features (MFCC) • Statistical model • Pattern classification technique • Hidden Markov model • Performance is strongly tied to selected features and statistical model • e.g., Khan et al., 2009; Cavel et al., 2005; Valenzise, 2007; Dufaux, 2000;

41. Acoustic Signal Classification - Typical Approach Other algorithm specific parameters Frame length Classifier parameters Different feature sets 67%-87% Ntalampiras et al., 2009, EURASIP Journal on Advances in Signal Processing

42. A Statistical Model for Time-Series • Acoustic signals can be distinguished by their time-frequency structure • Model an acoustic signal as a collection of spectral and energy components • Collection of AR models • How many? • What AR order? (Spectral complexity) • Resulting models must ultimately provide a means of adapting with new information • Nonparametric Bayesian methods provide a principled way to infer model complexity without explicitly enumerating each model • Uncertain-order AR models • Dirichlet process • Variational Bayesian inference provides a parameterized posterior density to enable computationally tractable updating Muzzle Blast Glass Breaking Door Slam

43. Uncertain-Order AR Models (Morton, 2011) • AR models • AR order (number of previous samples) controls the spectral complexity • Poles in the complex plane • Over estimation results in over fitting • Especially in data limit scenarios • Bayesian Inference • Conjugate prior (computationally simple) • Consider AR order as a random parameter • Infer correct order from data

44. Nonparametric Bayesian Inference Nonparametric ≠ no parameters Nonparametric = number of parameters determined by the data • Dirichlet Process (Ferguson, 1973) • Probability density function for probability density functions • A draw from a DP is an infinite discrete density • Draw from G - Chinese restaurant process 44

45. Stick Breaking HMM (Paisley, 2009) • Utilize standard HMM structure but • Model each row of the transition matrix with a stick-breaking construction • Technically truncated to a maximum value “T” (set as high as computationally possible) • Must use Bayesian inference • Model stick proportions with beta densities - [0, 1] • The use of states is penalized to promote automated selection AR Model Parameters HMM

46. UOAR SBHMM • The density within each state in the SBHMM is AR and uses an UOAR model prior • Parameter Inference • Determines the number of unique spectral and energy components • Determines the spectral complexity of each component • Models occurrence of the components using a Markov assumption • VB approximation enables tractable recursive Bayesian inference • Similar algorithm to standard EM forward backwards • Assume S maximum states (set arbitrarily high) Initial statestick-breaking Transition stick-breaking UOAR models for each state 46

47. UOAR SBHMM - Example • Synthetic data • 2 States • AR length 4 • AR length 6 • Number of states and AR orders are correctly determined • Synthetic PSD closely matches the STFT 47

48. UOAR SBHMM - Acoustic Signal Modeling • Muzzle blast from a Glock Model 17 • Model inferred from 10 examples • Maximum number of states S=25 • 6 states utilized • Most AR components have a low order • (2 or 4) • Broad spectral range • UOAR SBHMM characterizes time-frequency information similar to the STFT 48

49. UOAR SBHMMGeneration of Synthetic Acoustic Signals • UOAR SBHMM is a generative model for time domain data and can be used to generate signals with similar time-frequency characteristics • Spectral information and complexity closely match original signal • Markov assumption does not perfectly model occurrence of the components Bird Chirp Muzzle Blast 49

50. UOAR SBHMM Gunshot DetectionFalse Alarm Rejection • UOAR SBHMM used to model each class • Glass breaking, Door slam, Wood smashing, Muzzle blasts • 93.75% Correct • Feature based classification • 12 Characterizing features • Relevance Vector Machine • 85% Correct • Tuning may improve application specific performance 50