Part I: Classifier Performance

1. Part I: Classifier Performance Mahesan Niranjan Department of Computer Science The University of Sheffield M.Niranjan@Sheffield.ac.uk & Cambridge Bioinformatics Limited Mahesan.Niranjan@ntlworld.com

2. Relevant Reading • Bishop, Neural Networks for Pattern Recognition • http://www.ncrg.aston.ac.uk/netlab • David Hand, Construction and Assessment of Classification Rules • Lovell, et. Al. CUED/F-INFENG/TR.299 • Scott et al CUED/F-INFENG/TR.323 reports linked from http://www.dcs.shef.ac.uk/~niranjan Mahesan Niranjan

3. Pattern Recognition Framework Mahesan Niranjan

4. Two Approaches to Pattern Recognition • Probabilistic via explicit modelling of probabilities encountered in Bayes’ formula • Parametric form for class boundary and optimise it • In some specific cases (often not) both reduce to the same answer Mahesan Niranjan

5. Pattern Recognition: Simple case O • Gaussian Distributions • Isotropic • Equal Variances • Optimal Classifier: • Distance to mean • Linear Class Boundary Mahesan Niranjan

6. Mahalanobis Distance Distance can be misleading O Optimal Classifier for this case is Fisher Linear Discriminant Mahesan Niranjan

7. X X X X X X X X X X O X X O O O O O O O O O Support Vector MachinesMaximum Margin Perceptron Mahesan Niranjan

8. X X X X X X X O O X O O O O X O O X O O O O O O Support Vector MachinesNonlinear Kernel Functions Mahesan Niranjan

9. Support Vector MachinesComputations • Quadratic Programming • Class boundary defined only by data that lie close to it - support vectors • Kernels in data space equal scalar products in higher dimensional space Mahesan Niranjan

10. Support Vector MachinesThe Hypes • Strong theoretical basis - Computational Learning Theory; complexity controlled by the Vapnik-Chervonenkis dimension • Not many parameters to tune • High performance on many practical problems, high dimensional problems in particular Mahesan Niranjan

11. Support Vector MachinesThe Truths • Worst case bounds from Learning theory are not very practical • Several parameters to tune • What kernel? • Internal workings of the optimiser • Noise in training data • Performance? • depends on who you ask Mahesan Niranjan

12. SVM: data driven kernel • Fisher Kernel [Jaakola & Haussler] • Kernel based on a generative model of all the data Mahesan Niranjan

13. Classifier Performance • Error rates can be misleading • Imbalance in training/test data • 98% of population healthy • 2% population has disease • Cost of misclassification can change after design of classifier Mahesan Niranjan

14. Adverse Outcome x Benign Outcome x x Class Boundary x x x x x x x x x Threshold Mahesan Niranjan

15. True Positive False Positive Area under the ROC Curve: Neat Statistical Interpretation Mahesan Niranjan

16. Convex Hull of ROC Curves True Positive False Positive Mahesan Niranjan

17. Yeast Gene Example: MATLAB Demo here Mahesan Niranjan

18. Part II: Particle Filters for Tracking and Sequential Problems Mahesan Niranjan Department of Computer Science The University of Sheffield

19. Overview • Motivation • State Space Model • Kalman Filter and Extensions • Sequential MCMC Methods • Particle Filter & Variants Mahesan Niranjan

20. Motivation • Neural Networks for Learning: • Function Approximation • Statistical Estimation • Dynamical Systems • Parallel Processing • Guarantee Generalisation: • Regularise / control complexity • Cross validate to detect / avoid overfitting • Bootstrap to deal with model / data uncertainty • Many of the above tricks won’t work in a sequential setting Mahesan Niranjan

21. Interesting Applications • Speech Signal Processing • Medical Signals • Monitoring Liver Transplant Patients • Tracking the prices of Options contracts in computational finance Mahesan Niranjan

22. Good References • Bar-Shalom and Fortman: Tracking and Data Association • Jazwinski: Stochastic Processes and Filtering Theory • Arulampalam et al: “Tutorial on Particle Filters…”; IEEE Transactions on Signal Processing • Arnaud Doucet: Technical Report 310, Cambridge University Engineering Department • Benveniste, A et al: Adaptive Algorithms and Stochastic Approximation • Simon Haykin: Adaptive Filters Mahesan Niranjan

23. Matrix Inversion Lemma Mahesan Niranjan

24. Linear Regression Mahesan Niranjan

25. Recursive Least Squares Mahesan Niranjan

26. State Process Noise Measurement Noise Observation State Space Model Mahesan Niranjan

27. Simple Linear Gaussian Model Mahesan Niranjan

28. Prediction Correction Kalman Filter Mahesan Niranjan

29. Innovation Kalman Gain Kalman Filter Mahesan Niranjan

30. Prior Likelihood Innovation Probability Bayesian Setting • Run Multiple Models and Switch - Bar-Shalom • Set Noise Levels to Max Likelihood Values - Jazwinski Mahesan Niranjan

31. Extended Kalman Filter Taylor Series Expansion around the operating point First Order Second Order Iterated Extended Kalman Filter Lee Feldkamp @ Ford Successful training of Recurrent Neural Networks Mahesan Niranjan

32. Iterated Extended Kalman Filter Local Linearization of State and / or Observation Propagation and Update Mahesan Niranjan

33. Generate some points at time Unscented Kalman Filter So they can represent the mean and covariance: Propagate these through the state equations Recompute predicted mean and covariance: Mahesan Niranjan

34. Recompute: Recipe to define: Mahesan Niranjan

35. Formant Tracking Example Excitation Speech Linear Filter Mahesan Niranjan

36. Formant Tracking Example Mahesan Niranjan

37. Formant Track Example Mahesan Niranjan

38. Grid-based methods Discretize continuous state into “cells” Integrating probabilities over each partition Fixed partitioning of state space  Mahesan Niranjan

39. Parameters Uncertainty over parameters Sampling Methods: Bayesian Inference Inference: Mahesan Niranjan

40. Basic Tool: Composition [Tanner] To generate samples of Mahesan Niranjan

41. Importance Sampling Mahesan Niranjan

42. Particle Filters Bootstrap Filters ( Gordon et al, Tracking ) CONDENSATION Algorithm ( Isard et al, Vision ) Prediction Weights of Sample Mahesan Niranjan

43. Sequential Importance Sampling Recursive update of weights Only upto a constant of proportionality Mahesan Niranjan

44. Effective number of particles Degeneracy in SIS Variance of weights monotonically increases  All except one decay to zero very rapidly Resample if Mahesan Niranjan

45. Sampling, Importance Re-Sampling (SIR) Multiply samples of high weight; kill off samples in parts of space not relevant  “Particle Collapse” Mahesan Niranjan

46. Suppose Sample with respect to Rao-Blackwell Integrate with respect to Marginalizing Part of the State Space Possible to analytically integrate with respect to part of the state space Mahesan Niranjan

47. Variations to the Basic Algorithm • Integrate out part of the state space • Rao-Blackwellized particle filters ( e.g. Multi-layer perceptron with linear output layer ) • Variational Importance Sampling ( Lawrence et al ) • Auxilliary Particle Filters ( Pitt et al ) • Regularized Particle Filters • Likelihood Particle Filters Mahesan Niranjan

48. Regularised PF: basic idea Samples Kernel Density Resample Propagate in time Mahesan Niranjan

49. Conclusion / Summary • Collection of powerful algorithms • New and interesting signal processing problems Mahesan Niranjan