Pattern Recognition: Statistical and Neural

1. Nanjing University of Science & Technology Pattern Recognition:Statistical and Neural Lonnie C. Ludeman Lecture 19 Oct 26, 2005

2. Lecture 19 Topics 1. Structures of Optimal Statistical Classifiers 2. Neural Network History 3. Biological Neural Networks 4. Modified McColloch Pitts Model – Example 5. Artificial Neural Element - Definition

3. Structures from Statistical Decision Theory and Perceptron Algorithm (a). Two Class Likelihood Ratio test (b) Structures to Motivate Neural Networks (c) Two Class General Linear (d). N Class General minimum P(error) (e) N Class Gaussian

4. Decision Rule if l(x) > T decide x from C1 < T decide x from C2 = T decide x randomly between C1 and C2

5. C1 if f(x) > T < C2 Decision Rule

6. C1 if f(x) > T < C2 Decision Rule

7. if p(x | Ck) > p(x | Cj ) for all j ≠ k decide Ck Decision Rule

8. (e) M Class Gaussian

9. These Structures will be seen to be similar to the structures used in designs with Neural Networks

10. Important Events in Neural Networks History 1943McColloch Pitts Model 1958Perceptron Algorithm- Rosenblatt 1960-62ADALINE – Widrow and Hoff 1969Minsky and Papert-Limitations 1980’sGrossberg , Hopfield, and Rumelhart – Backpropagation algorithms, Adaptive Resonance Theory Period of Revival 1990’s Maturation of Field

11. IMPORTANT BOOKS 1990Artificial Neural Systems- Jacek M. Zurada 1992Neural Networks and Fuzzy Systems- Bart Kosco 1994 Neural Networks: A Comprehensive Foundation- Simon Haykin

12. Neural Networks Biological (Real) Mathematical (Artificial) How do you tell them apart ??? Squeeze them !!! Excite them !!!

13. Biological Neuron

14. Action Potential

16. Biological Neural Network

17. End Bulb Connection

18. Integration at axon-dendrite junction

19. Modified McColloch Pitts Neuronal Model (Threshold Logic Unit (TLU) )

20. Question What can we do with a Modified McColloch-Pitts Model ??? Answer Surprisingly we can model all logical expressions !!!

21. Implementation of Logical AND and OR T = n - 1/2 T = 1/2

22. Implementation of Logical NOT -1/2 Since we can model logical AND, OR and NOT we can model alllogical expressions

23. Example – Implementation of a given logic expression Given: Implement f(x) using Modified McColloch-Pitts Neurons

24. Solution: -1/2 2.5 -1/2 T=1/2 -1/2 3.5 -1 -1/2

25. Artificial Neural Element (ANE) Node net Input Vector

26. Artificial Neural Element Mathematical Model net = w1x1 + w2x2 + …+ wnxn + wn+1 = wTx Linear Operation on x y = f(net) = f( wTx) Nonlinear Operation on x Nonlinear Activation Function

27. Artificial Neural Element Nodal Representation Vector Notation

28. Some Activation Functions

30. Hyperbolic Tangent Activation Function Definition Equivalent Form f(net)

31. Hyperbolic Activation Function

33. Non Monitonic Activation Functions can be very Useful Examples: Potential Function ApproachRadial Basis Functions

34. Summary Lecture 19 1. Structures of Optimal Statistical Classifiers 2. Neural Network History 3. Biological Neural Networks 4. Modified McColloch Pitts Model – Example 5. Artificial Neural Element - Definition

35. End of Lecture 19