Fundamentals of Neural Network Theory

1. NEURAL NETWORK THEORY NEURAL DYNAMIC1: ACTIVATIONHS AND SIGNALS

2. MAIN POINTS: • NEURONS AS FUNCTIONS(神经元函数) • SIGNAL MONOTONICITY（信号单调性） • BIOLOGICAL ACTIVATIONS AND SIGNALS（生物激励与信号） • NEURON FIELDS（神经域） • NEURONAL DYNAMICAL SYSTEMS（神经诊断系统） • COMMON SIGNAL FUNCTION（一般信号方程） • PULSE-CODED SIGNAL FUNCTION（脉冲编码信号方程）

3. NEURONS AS FUNCTION , Figure 1. Neuron Structure Model Relationship of input-output:

4. NEURONS AS FUNCTION • Common nonlinear transduction description: a sigmoidal or S-shaped curve Fig.2 s(x) is a bounded monotone-nondecreasing function of x Signal Function: Neurons transduce an unbounded input activation x(t) at time t into a bounded output signal S(x(t)).

5. SIGNAL MONOTONICITY • In general, signal functions are monotone nondecreasing S’>=0. In practice this means signal functions have an upper bound or saturation value. • An important exception: bell-shaped signal function or Gaussian signal functions The sign of the signal-activation derivation s’ is opposite the sign of the activation x. We shall assume signal functions are monotone nondecreasing unless stated otherwise.

6. SIGNAL MONOTONICITY • Generalized Gaussian signal function define potential or radial basis function: input activation vector: : variance: mean vector: Because the function depend on all neuronal activations not just the ith activation, we shall consider only scalar-input signal functions:

7. SIGNAL MONOTONICITY • A property of signal monotonicity: semi-linearity • Comparation: • Linear signal functions: • computation and analysis is comparatively easy; do not suppress noise. b. Nonlinear signal functions: Increases a network’s computational richness and facilitates noise suppression; risks computational and analytical intractability;

8. SIGNAL MONOTONICITY • Signal and activation velocities the signal velocity: =dS/dt Signal velocities depend explicitly on action velocities. This dependence will increase the number of unsupervised learning laws.

9. BIOLOGICAL ACTIVATIONS AND SIGNALS • Introduction to units ： Dendrite: input Axon: output Synapse: transduce signal Membrane: potential difference between inside and outside of neuron Fig3. Key functional units of a biological neuron

10. BIOLOGICAL ACTIVATIONS AND SIGNALS • Competitive Neuronal Signal Signal values are usually binary and bipolar. Binary signal functions : Bipolar signal functions :

11. NEURON FIELDS In general, neural networks contain many fields of neurons. Neurons within a field are topological. Denotation: : input field : output field Neural system samples the function m times to generate the associated pairs • Classification: Zeroth-order topological (simplest) Three-dimensional and volume topological (complex)

12. NEURONAL DYNAMICAL SYSTEMS • Description: A system of first-order differential or difference equations that govern the time evolution of the neuronal activations or membrane potentials Activation differential equations: denote the activation time functions of the ith neuron in and jth neuron in • Classification: Automomous systems: activations are independent of t Nonautonomous systems: depend on t

13. NEURONAL DYNAMICAL SYSTEMS • Neuronal State spaces So the state space of the entire neuronal dynamical system is: Augmentation : Concatenate fields have different computational, metrical or other properties

14. NEURONAL DYNAMICAL SYSTEMS Signal state spaces as hypercubes Fig.4 Neural and fuzzy computations conincide.

15. NEURONAL DYNAMICAL SYSTEMS • Neuronal activations as short-term memory Short-term memory(STM) : activation Long-term memory(LTM) : synapse

16. S k x o COMMON SIGNAL FUNCTION 1、Liner Function S(x) = cx + k , c>0

17. S r -θ θ x -r COMMON SIGNAL FUNCTION 2. Ramp Function r if x≥θ S(x)= cx if |x|<θ -r if x≤-θ r>0, r is a constant.

18. COMMON SIGNAL FUNCTION 3、threshold linear signal function: a special Ramp Function Another form:

19. COMMON SIGNAL FUNCTION 4、logistic signal function: Where c>0. So the logistic signal function is monotone increasing.

20. COMMON SIGNAL FUNCTION 5、threshold signal function: Where T is an arbitrary real-valued threshold,and k indicates the discrete time step.

21. COMMON SIGNAL FUNCTION 6、hyperbolic-tangent signal function: Another form:

22. COMMON SIGNAL FUNCTION 7、threshold exponential signal function: When

23. COMMON SIGNAL FUNCTION 8、exponential-distribution signal function: When

24. COMMON SIGNAL FUNCTION 9、the family of ratio-polynomial signal function: An example For

25. PULSE-CODED SIGNAL FUNCTION • Description: • Pulse trains arriving in a sampling interval seems to be the bearer of neuronal signal information. Pulse-coded formulation: where denote binary pulse functions that summarize the excitation of membrane potential.

26. PULSE-CODED SIGNAL FUNCTION • Velocity-difference property of pulse-coded signals A simple form for the signal velocity: Current pulse and current signal or expected pulse frequency are available quantities. Another computational advantage: If

27. 欢迎大家提出意见建议！ 常虹 2006.9.25