Unsupervised Modelling of ‘Usual’ Events and Detecting Anomalies in Videos

1. Unsupervised Modelling of ‘Usual’ Events and Detecting Anomalies in Videos Advisor: AmitabhaMukerjee Deepak Pathak (10222) AbhijitSharang (10007)

2. 1. Motivation • Large volume of un-annotated video data • Surveillance cameras • Automatic behaviour learning Present focus: • Traffic Dataset containing abnormal events [Varadrajan, 2009]

3. 2. “Anomaly” • Anomaly refers to the unusual or rare event occurring in the video • Definition is ambiguous and depends on context Idea: • Learn the “usual” events in the video and use the information to tag the rare events.

4. 3.1.Overview of Approach • Model the “usual” behaviour of scene using parametric bayesian model • Reconstruct the new behaviour from estimated parametric model • Threshold on the similarity measure of reconstructed and actual distribution

5. 3.2.Topic Modelling • Given: Document and Vocabulary * Document is histogram over vocabulary • Goal: Identify topics in a given set of Documents Idea: Topics are latent variables Alternate view : • Clustering in topic space • Dimensionality reduction

6. 3.3.Models in practice • LSA Non-parametric clustering into topics using SVD. • pLSA : Learns probability distribution over fixed number of topics; Graphical model based approach. • LDA : Extension of pLSA with dirichlet prior for topic distribution. Fully generative model.

7. 4. Vision to NLP : Notations

8. 5.1.Video Clips • 45 minute video footage of traffic available • 25 frames per second • 4 kinds of anomaly • Divided into clips of fixed size of say 4 – 6 seconds

9. 5.2.Visual Words • Each frame is 288 x 360 • Frame is divided into 15 x 18 parts, each part containing 400 pixels • Features • Optical flow • Object size • Background subtraction is performed on each frame to obtain the objects in foreground. Features are computed only for these objects • Foreground objects consist of vehicles, pedestrians and cyclists

10. 5.2.Visual Words (contd..) • Foreground pixels then divided into “big” and “small” blobs (connected components) • Optical flow computed on foreground • Flow vector quantised into 5 values : • Static • Dynamic- up, down, left and right • 15x28x5x2 different “words” obtained

11. 5.3.Foreground Extraction

12. 5.4. Optical Flow: Heat Map

13. 6. Modelling pLSA • Training Dataset: no or very less “anomaly” • Test Dataset: usual + anomalous events Procedure • Learn and from training data • Keeping , estimate on test data • Threshold on likelihood estimate of individual test video clips

14. 7. Results Demo • 3 clips • 3 different types of anomalies

15. 8. Action Topics Extracted(histogram over topics on x axis – 20 topics)

16. 8. Action Topics Extracted • Demo video clips

17. 8. Results (ROC plot)

18. 8. Results (PR curve)

19. 9. Current Limitations • Too much context-dependency modelling. Specific feature design. • Full video clip is declared anomalous.

20. 9. Future work • Generalised words from HoG and HoF • Localisation of anomaly • Generative model for topics

21. References • Varadarajan, Jagannadan, and J-M. Odobez. "Topic models for scene analysis and abnormality detection." Computer Vision Workshops (ICCV Workshops), 2009 IEEE 12th International Conference on. IEEE, 2009. • Niebles, Juan Carlos, Hongcheng Wang, and Li Fei-Fei. "Unsupervised learning of human action categories using spatial-temporal words." International Journal of Computer Vision 79.3 (2008): 299-318. • Mahadevan, Vijay, et al. "Anomaly detection in crowded scenes." Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on. IEEE, 2010. • Roshtkhari, MehrsanJavan, and Martin D. Levine. "Online Dominant and Anomalous Behavior Detection in Videos.“ • Farneback, Gunnar. "Fast and accurate motion estimation using orientation tensors and parametric motion models." Pattern Recognition, 2000. Proceedings. 15th International Conference on. Vol. 1. IEEE, 2000. • Hofmann, Thomas. "Probabilistic latent semantic indexing." Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval. ACM, 1999. • Blei, David M., Andrew Y. Ng, and Michael I. Jordan. "Latent dirichlet allocation." the Journal of machine Learning research 3 (2003): 993-1022.

22. Extra Slides • About • Background subtraction • Optical Flow • pLSA and its EM • Normalized Likelihood

23. Background subtraction • Extraction of foreground from image • Frame difference • D(t+1) = | I(x,y,t+1) – I(x,y,t) | • Thresholding on the value to get a binary output • Simplistic approach(can do with extra data but cannot miss any essential element) • Foreground smoothened using median filter

24. Optical flow example (a) Translation perpendicular to a surface. (b) Rotation about axis perpendicular to image plane. (c) Translation parallel to a surface at a constant distance. (d) Translation parallel to an obstacle in front of a more distant background. Slides from Apratim Sharma’s presentation on optical flow,CS676

25. Optical flow mathematics • Gradient based optical flow • Basic assumption: • I(x+Δx,y+Δy,t+Δt) = I(x,y,t) • Expanded to get IxVx+IyVy+It = 0 • Sparse flow or dense flow • Dense flow constraint: • Smoothness : motion vectors are spatially smooth • Minimise a global energy function

26. pLSA • Fixed number of topics : {z1, z2 , z3 ,…,zk }.Each word in the vocabulary is attached with a single topic. • Topics are hidden variables. Used for modelling the probability distribution • Computation • Marginalise over hidden variables • Conditional independence assumption: p(w|z) and p(d|z) are independent of each other

27. EM Algorithm: Intuition • E-Step • Expectation step where expectation of the likelihood function is calculated with the current parameter values • M-Step • Update the parameters with the calculated posterior probabilities • Find the parameters that maximizes the likelihood function

28. EM: Formalism

29. EM in pLSA: E Step • It is the probability that a word w occurring in a document d, is explained by aspect z (based on some calculations)

30. EM in pLSA: M Step • All these equations use p(z|d,w) calculated in E Step • Converges to local maximum of the likelihood function

31. Anomaly Detection: Likelihood

32. Normalized Likelihood: Threshold • Normalized likelihood measure is calculated as follows -