Artificial Neural Networks

1. Artificial Neural Networks Kong Da, Xueyu Lei & Paul McKay

2. Neural Networks

3. What is Neural Networks • An Artificial Neural Network is a computational simulation of a biological neural network. • composed of a large number of highly interconnected processing elements(neurons) • an information processing paradigm • learn by example

4. What is Neural Networks • Inspiration from brain Figure1 Neuron structure Figure 2 ANN structure Figure1: http://www.studyblue.com/notes/note/n/biological-foundations-neuron-communication-/deck/1025438 Figure2: http://www.gdl.cinvestav.mx/~edb/students/evazquez/index.html

5. History of ANNs Warren McCulloch and Walter Pitts modeled a simple neural network with electrical circuits Bernard Widrow and Marcian Hoff of Stanford developed models called "ADALINE" and "MADALINE." Donald Hebb pointed out that neural pathways are strengthened each time that they are used in The Organization of Behavior The IBM research laboratories led the first effort to simulate a neural network Bayesian network Learning Vector Quantization Long short term memory network physical neural network Hierarchical temporal memory (HTM) …… 1943 • The first recurrent network 1949 • ADALINE Could predict the next bit • MADALINE The first neural network applied to a real world problem 1950’s 1959  "deep learning" gained traction in the mid-2000s after a publication by Geoffrey Hinton and RuslanSalakhutdinov 1980 1982 Convolutional neural networks were introduced in a 1980 paper by KunihikoFukushima John Hopfield of Caltech created more useful machines by using bidirectional lines (Hopfield Network) 1985 Companies are working on three types of neuro chips - digital, analog, and optical A Boltzmann machine,a type of stochastic recurrent neural network is invented by Geoffrey Hinton and Terry Sejnowski Mid 2000s NOW

6. History of ANNs • Figure 3 MADALINE structure Figure 4 An example Boltzmann machine Figure 5 a maximally simple LSTM network • Figure3:http://www.drdobbs.com/the-foundation-of-neural-networks-the-ad/184402585 • Figure4:http://en.wikipedia.org/wiki/Boltzmann_machine • Figure5: http://www.schraudolph.org/teach/NNcourse/lstm.html • Figure 6 Neural chip http://www.gizmag.com/neuromorphic-chips/28586/

7. Types of ANNs

8. Perceptron

9. Activation function WikiBooks.Artificial Neural Networks/Activation Functions. 25 August 2014.

10. Example 1 Çelebi, ÖmerCengiz. Neural Networks and Pattern Recognition Using MATLAB. Retrieved 25 August 2014.

11. Example 2 Çelebi, ÖmerCengiz. Neural Networks and Pattern Recognition Using MATLAB. Retrieved 25 August 2014.

12. Neural network topology • Multilayer • Interconnected • Feed forward/recurrent • Deep learning

13. Training • Back propagation • Learning rate • Batch learning • Stochastic gradient descent • Evolutionary algorithms Reference: LeCun et al. Efficient BackProp. 1998.

14. Application areas Function approximation Classification http://www.eweb.unex.es/eweb/fisteor/santos/sby.html Screenshot from https://www.youtube.com/watch?v=KuPai0ogiHk

15. Application areas Control Data processing. http://www.seit.adfa.edu.au/research/details2.php?page_id=410&topic=Adaptive_Flight_Control http://www.cslu.ogi.edu/tutordemos/nnet_recog/recog.html

16. Application areas Robotics Screenshots from https://www.youtube.com/watch?v=V2ADU8YWIug