Artificial Neural Networks

Technological Educational Institute Of Crete Department Of Applied Informatics and Multimedia Intelligent Systems Laboratory Artificial Neural Networks Prof. George Papadourakis, Ph.D.

Introduction to Neural Networks “An Artificial Neural Network is a mathematical model inspired by biological neural networks”

Introduction to Neural Networks • Artificial Neural Networksare models of the brain and the nervous system, designed to process information more like thebrainthan aserialcomputer. • Due to their brain-like design, ANNs are very powerful problemsolvingtools. • They have very simpleprinciplesyet.. • ..very complex behaviors. Much like your own brain!

Introduction to Neural Networks Introduction to Neural Networks Comparison Conventional Computing ANNs

Introduction to Neural Networks Conventional Computing in more detail : • Deductive Reasoning. • We apply known rules to input data to produce output. • Computation is centralized, synchronous, and serial. • Memory is packeted, literally stored, and location addressable. • Not fault tolerant. One transistor goes and it no longer works. • Exact. • Static connectivity. • Applicable to problems where the rules are well defined and the data precise. for example.. 30! = ..

Introduction to Neural Networks Artificial Neural Networks in more detail : • Inductive Reasoning. Given the input and output data (training examples), we construct the rules. • Computation is collective, asynchronous, and parallel. • Memory is distributed, internalized, short term and content addressable. • Fault tolerant, redundancy, and sharing of responsibilities. • Inexact. • Dynamic connectivity. • Applicable ifrules are unknown or complicated, or if data are noisy or partial. for example.. Weather prediction!

Introduction to Neural Networks General ANN applications : • Assessment Engineering: Product inspection monitoring and control Defence: Target tracking Security: Motion detection, camera surveillance, fingerprint matching • Forecasting and prediction Finance: Foreign exchange rate and stock market forecasting Agriculture: Crop yield forecasting Marketing: Sales forecasting Meteorology: Weather prediction • Classification Marketing: Consumer spending pattern classification Defense: Radar and sonar image classification Agriculture & fishing: Fruit and catch grading Medicine: Ultrasound/electrocardiogram image classification, EEG • Recognition and identification General computingandtelecommunications: Speech, vision andhandwriting recognition Finance: Signature verification and bank note

Introduction to Neural Networks ANN applications in science : • ComputerScientistswant to find out about the properties of non-symbolic information processing with Neural Nets and about learning systems in general. • Statisticiansuse neural nets as flexible, non-linear regression and classification models. • Engineersexploit the capabilities of Neural Networks in many areas, such as signal processing and automatic control. • CognitiveScientistsview Neural Networks as a possible apparatus to describe models of thinking and consciousness. • Neuro-Physiologistsuse NNs to describe and explore medium-level brain function (e.g. memory) . • Physicistsuse them to model phenomena in statistical mechanics and for a lot of other tasks. • Biologistsuse Neural Networks to interpret nucleotide sequences. • Philosophersandmanymorebenefit from Neural Networks' brain-based design.

Introduction to Neural Networks ANNs' Strenghts : • They do almost everything a normal digital computer can do. Almost any mapping between vector spaces can be approximated to arbitrary precision by feed-forward NNs. • They are especially useful for classificationandfunctionapproximationproblems usually when rules such as those that might be used in an expert system cannot be easily applied. ANNs' Weaknesses : • NNs are, at least today, difficult to successfully apply to problems concerning manipulationofsymbolsand memory. And there are no Neural Network training methods that can magically create information that is not contained in the training data.

Introduction to Neural Networks Biological Neural Networks Pigeonsas art experts(Watanabe et al. 1995) • Question :Can we teach this pigeon to recognise a specific artist's work?

Introduction to Neural Networks Experiment: • Pigeon in a Skinner box. • Present paintings of two different artists (e.g. Chagall / Van Gogh). • Reward for pecking when presented a particular artist (e.g. Van Gogh). ?? ??

Introduction to Neural Networks • Results • Pigeons were able to discriminate between Van Gogh and Chagall with 95% accuracy when presented with pictures they had been trained on. • Discrimination still 85% successful for previously unseenpaintings of the artists! • That means the pigeons do not simply memorise the pictures. They extractedand recognisedpatterns (the artist's style)and generalisedfrom the already seen to make predictions. • This is what Neural Networks (biological and artificial) are good at, unlike conventional computers.

Introduction to Neural Networks TheBrain • A collection of about 10 billion interconnected neurons, each neuron cell • using biochemical reactions toreceive, process and transmit information. • Each terminal button (end of A), the far end of the neuron that releases • chemicals, is connected to other neurons across a small gap called a synapse. Neuron Synapse

Introduction to Neural Networks Firing a Neuron • A neuron's dendritic tree is connected to thousands of neighbouring neurons. When one of those neuronsfire, a positive or negative charge is received by one of the dendrites. The strengths of all the received charges are added together through the processes of spatial and temporal summation.

Introduction to Neural Networks Intelligence The human brain is an extremely powerful computer. So why haven’t we managed to recreate it artificially yet? • Hard to comprehend the fluidity and adaptability of the information processing done inside the brain. • Not powerful enough programming languages. • Human brain architecture better for natural information processing. • The brain is too complex and powerful to mimic. • ..and many more.

Introduction to Neural Networks Intelligence Example : It’s easy for your brain to understand the words “THE CAT”, yet the very same thing is extremely hard for a computer. The idea behind CAPTCHA (“Completely Automated Public Turing test to tell Computers and Humans Apart”) is based on this very fact.

Introduction to Neural Networks Artificial NNs, the Basics One way to make computers more like our brain is a Parallel and Distributed Processing approach : • A lot of simple processing units are used instead of a centralized one. • They are connected to and affect each other. • In these connections is where the information is stored. • A single unit does not stand for any concept in particular.

Inputs Weights w p 1 1 a w 2 f p 2 w Output 3 p 3 1 Bias ( ) ( ) å = + + + = + Output f p w p w p w b f w p bias i i 1 1 2 2 3 3 Introduction to Neural Networks Artificial NNs, the Basics ANNs incorporate the two fundamental components of biological neural nets, Neurons andSynapses. Neurons Nodes Synapses Weights

Introduction to Neural Networks Introduction to Neural Networks Artificial NNs, the Basics Each neuron within the network is usually a simple processing unit which takes one or more inputs and produces an output. At each neuron, every input has an associated weight which modifies its strength. The neuron simply adds together all the inputs and calculates an output to be passed on. An Artificial Neural Network is composed of many artificial neurons that are linked together according to a specific network architecture. The objective of the Neural Network is to transform the inputs into meaningful outputs.

Introduction to Neural Networks Artificial NNs, Network Example Neural computing requires a number of neurons, to be connected together into a Neural Network. Neurons are arranged inlayers. Output = Σ Wipi + bias Weight Node

1.0 0.5 0.7 W1=0.32 W2=0.46 W3=0.81 Output = (1.0x0.32) + (0.5x0.46) + (0.7x0.81) = 1.117 Introduction to Neural Networks Artificial NNs, Output Example Then, the output enters an activation function to determine its final value. • For this example, we limit the output between 0-1 range. Output = Σ Wipi + bias 1 = 0.466 So the node’s final output is 0.466 1.117 1+e

Introduction to Neural Networks Artificial NNs, Input Examples • Data is presented to the network in the input layer. • Depending on the problem, inputs can be anything.. Examples are pixel intensity (for object recognition problems), molecule concentrations (making an artificial nose), share prices (for stock market prediction). • Data usually requires pre-processing. Its a lot harder for the network to handle inputs with huge differences from each other (0.00001 and 10,000,000). We can make the network's life a lot easier, thus getting better results, if we limit the input in small, predictable ranges. • We can represent more complex data, like names of diseases by assigning them to bits or numbers. (e.g. 001 = “Pancreatic Cancer”, 010 = “Liver Cancer”).

Introduction to Neural Networks Artificial NNs, Activation Functions • Activation functions describe the output behaviour of the neurons. They are, generally, not linear since linear functions are proportional to the input, thus limited.

Introduction to Neural Networks • Artificial NNs, Architecture With the term architecturewe mean the number of neuron layers and neurons per layer we use. Single Layer, 3 : 1 Architecture Multi-Layer, 2 : 4 : 1 Architecture

Introduction to Neural Networks • Artificial NNs, Architecture Hidden layers, the layers in between theinput and outputlayers, are the place where the actual processing is done. The core of a Neural Network. More than one hidden layers can be used. Multi-layered is more powerful than single layeredarchitecture.

Introduction to Neural Networks • Artificial NNs, Feed-Forward Networks Feed-forwardare the networks that allow information to travel only one way, from input to output, as opposed to feedback providing networks.Feed-forward ANNs tend to be straight forward networks that associate inputs with outputs. They are extensively used in pattern recognition. This type of organizationis also referred to as bottom-up or top-down. Information Flow Top – down organization

Introduction to Neural Networks • Artificial NNs, Feedback Networks Feedback networkscan have information travelling in bothdirectionsby introducing loopsin the network. Feedbacknetworksare very powerfuland can get extremely complicated. They are dynamic; their state is changing continuously until they reach an equilibriumpoint. They remain at the equilibrium point until the inputchangesand a new equilibrium needs to be found.

Introduction to Neural Networks • Artificial NNs, Network Training Methods “Artificial Neural Networks learn! They are not hard-coded. They change themselves. They adapt.”

Introduction to Neural Networks • Artificial NNs, Network Training Methods • Supervised Learning. • Both the inputs and the outputs are provided. Then, the network processes the inputs • and compares its resulting outputs against the desired outputs. Errors made are, then, • back-propagated through the system, causing the weights that control the network to • slightly change in a benefitting way. • Supervised learning needs aTraining Set to enable the above error detection. During the training of a network the same set of data is processed many times as the weights change/improve again and again. This change in the weights is the network “adapting” to the specific problem. Example : Multilayer perceptron architecture.

Introduction to Neural Networks • Artificial NNs, Network Training Methods • Unsupervised Learning. A Training Set (desired outputs) is not needed. The system itself must decide what features it will use to group the input data. This is often referred to as self-organization or adaption. Examples : Kohonen Self Organizing Maps Neural Gas Art Map

Introduction to Neural Networks • Artificial NNs, ANN Example Optical Character Recognition • Task : Learn to discriminate between two different typed characters. • Data : • Characters: • Letter ‘A’ • Letter ‘b’ • Format • Image pixel values (binary) • Analogy : Eye - Optical Nerve - Brain

Introduction to Neural Networks • Artificial NNs, ANN Example Network architecture : • Feed forward network: • 20 input nodes (one for each pixel value) • 6 nodes in the hidden layer • 2 output nodes (0-1 for ‘A’, 1-0 for ‘b’)

Introduction to Neural Networks • Artificial NNs, ANN Example Presenting the results on an Untrained Network : = 1 = 0 0.43 0.26 20 Inputs 0.73 0.55 20 Inputs

Introduction to Neural Networks • Artificial NNs, ANN Example Calculating the Error : = 1 = 0 0.43 – 0 = 0.43 0.26 – 1 = 0.74 20 Inputs 0.73 – 1 = 0.27 0.55 – 0 = 0.55 20 Inputs

Introduction to Neural Networks • Artificial NNs, ANN Example Adjusting the Weights : = 1 = 0 0.43 – 0 = 0.43 0.26 – 0 = 0.74 20 Inputs 1.17 0.73 – 1 = 0.27 0.55 – 0 = 0.55 20 Inputs 0.82

Introduction to Neural Networks • Artificial NNs, ANN Repetition • The repetition process (sweep) for all training pairs: • Present data • Calculate error • Back-propagate error • Adjust weights • This process is repeated multiple times, until the error becomes small enough. Error Repetition (Epoch)

Introduction to Neural Networks • Artificial NNs, ANN Example Presenting the results on the, finally, Trained Network : = 1 = 0 0.01 0.99 20 Inputs 0.99 0.01 20 Inputs

Introduction to Neural Networks • Artificial NNs, Design Considerations • What transfer function should be used? • How many inputs does the network need? • How many hidden layers does the network need? • How many hidden neurons per hidden layer? • How many outputs should the network have? • There is no standard methodology to determine these values. • The answers to these questions emerge by a trial and error procedure. • This is one of the reasons why experience is an asset in ANNs.

Introduction to Neural Networks • Artificial NNs, Conclusion • Artificial Neural Networks are inspired by the learning processes that take place in biological systems. • Artificial neurons and neural networks try to imitate the working mechanisms of their biological counterparts. • Learning can be perceived as an optimisation process. • Biological neural learning happens by the modification of the synaptic strength (adjusting the weights). Artificial neural networks learn in the same way.

Introduction to Neural Networks • Artificial NNs, Historical Background The study of the human brain dates back thousands of years, but it has only been with the dawn of modern day electronics that man has begun to try and emulate the human brain and its thinking processes. The modern era of neural network research is credited with the work done by neuro-physiologist, Warren McCulloch and young mathematical prodigy WalterPitts in 1943. McCulloch had spent 20 years of life thinking about the "event" in the nervous system that allowed to us to think, feel ,etc.

Introduction to Neural Networks • Artificial NNs, Historical Background • Some NNs are models of biological neural networks and some are not, but historically, much of the inspiration for the field of NNs came from the desire to produce artificial systems capable of sophisticated, perhaps intelligent, computations similar to those that the human brain routinely performs, and thereby possibly to enhance our understanding of the human brain. • Most NNs have some sort of training rule. In other words, NNs learn from examples (as children learn to recognize dogs from examples of dogs) and exhibit some capability for generalization beyond the training data. • Neural computing must not be considered as a competitor to conventional computing. Rather, it should be seen as complementary as the most successful neural solutions have been those which operate in conjunctionwith existing, traditional techniques.

“If the brain were so simple that we • could understand it then we’d be • so simple that we couldn’t” • -Watson, Lyall

18th EAEEIE Conference Praha, Chech Republic Apo slides sta ellhnika pairnw kai vazw edw. H metafrash apo ta agglika slides.

Artificial Neural Networks