计算智能理论与应用

1. 计算智能理论与应用 复习提纲

2. 考试题型 • 1. 填空：20空，20分 • 2. 简答/计算：4题，80分

3. Intelligent Computing — Chapter 1 —Introduction 3

4. 课程内容 导论 神经计算 进化计算 群体智能 人工免疫算法 4 Spring 2012

5. 优化问题 优化技术？ 以数学为基础，解决各种工程问题优化解 优化技术的用途 系统控制 人工智能 模式识别 生产调度 5 Spring 2012

6. 优化问题 最优化问题的描述 最优化问题的数学模型的一般描述： 6 Spring 2012

7. 传统优化方法 待解决的问题 连续性问题，以微积分为基础，规模较小 传统的优化方法 理论上的准确与完美，主要方法：线性与非线性规划、动态规划、多目标规划、整数规划等；排队论、库存论、对策论、决策论等。 传统的评价方法 算法收敛性、收敛速度 7 Spring 2012

8. 现代优化方法 待解决的问题 离散性、不确定性、大规模 现代的优化方法 启发式算法（heuristic algorithm） 追求满意（近似解） 实用性强（解决实际工程问题） 现代的评价方法 算法复杂性 8 Spring 2012

9. 函数优化问题 令X为Rn上的有界子集（即变量的定义域）， 为n维实值函数，所谓函数f在X域上全局最小化就是寻求点x使得f(x)在X域上全局最小。 • 难点 • 高维 • 多峰值 9 Spring 2012

10. Intelligent Computing — Chapter 2 —Neural Computation 10

11. Gradient Descent

12. Algorithm: Gradient Descent

13. Newton Method

14. Algorithm: Newton Method

15. Categories of Neural Networks 按照结构分类 前向型 反馈型 按照学习方式分类 监督型 无监督型 信号类型方式 连续的 离散的 15

16. Perceptron w0 x0 = 1 x1 w1 Activation function Local Field v Input signal Output y x2 w2 Summing function Synaptic weights xm wm Error 16

17. Activation function 1.Linear function f(x)=ax 2. Step function 3. Ramp function 17

18. Activation function 4. Logistic function 5. Hyperbolic tangent 6. Gaussian function 18

19. Least Mean Square learning LMS = Least Mean Square learning Systems, more general than the previous perceptron learning rule. The concept is to minimize the total error, as measured over all training examples, P. O is the raw output, as calculated by We want to minimize the LMS: C-learning rate E W(old) W(new) W 19

20. Optimization 20

21. Optimization 思考：给一个数据集合， 如何训练感知机模型？ 21

22. BP Network Details Forward Pass: Error is calculated from outputs Used to update output weights Backward Pass: Error at hidden nodes is calculated by back propagating the error at the outputs through the new weights Hidden weights updated 22

23. BP Neural Networks Signal-flow graph highlighting the details of output neuron k connected to hidden neuron j. 23

24. BP Network Details 网络输出层第j个神经元的误差函数定义为: BP-1 而瞬间平方差函数E(n)，就是所有输出层神经元的平方差瞬间值之和： BP-2 其中集合 C是包含所有输出层神经元的子集合。 如果N为输入训练集中的元素个数,则均方差函数为: BP-3 24

25. BP Network Details 对于给定的一个训练集合,则可以用E(n)来代表网络学习此训练资料的性能指标.即网络训练的目标就是将E(n)最小化 第n次学习过程中,第j个神经元的输出通过以下方程计算: BP-4 BP-5 25

26. BP Network Details 权值wji(n) 的修正量 Δwji(n) 和梯度 E(n)/ wji(n)的值成正比关系。我们可将梯度表示为: 根据BP-4 我们定义 则权值wji(n) 的修正量 Δwji(n)就可以写为: 其中  是学习率参数, 所以可以根据以下公式来调整权值: 26

27. 1. 当第 j个神经元是输出层时 根据公式BP-1,BP-2得出: 27

28. 2. 当第j个神经元是隐含层时 j通过Wkj来链接k 如果k是输出层的神经元 隐含层的更新可以得出为 28

29. Activation function 从以上可以看出要使 (.) 微分值存在，激活函数连续的可微分的。通常,经常使用的激活函数为sigmoid函数: 29

30. Tutorial: Learning to Rank

31. Intelligent Computing — Chapter 3 —Evolutionary Computation 31

32. 种类 • 遗传算法 • 进化策略 • 遗传编程 • 差分进化 • 协同进化 ……

33. 遗传算法 • 表达（Representation） • 定长，变长 • 交叉，变异各是什么样的？ • 以背包问题为例

34. Binary Representation selected gene 1 2 3 4 5 6 7 chromosome 0 1 1 0 0 0 1 2 3 7 ratio of cj /wj 1.2 0.33 2.0 selected genes 1 1 1 descending order 7 2 3 1 1 1 7 output: items in knapsack 1 7 2 3 7 2 7 2 1 1 1 1 1 1 1 input: data set W =100 1. Select genes of xj=1 2.Sort items with the value xj = 1 in descending order by the ratio of profit to weight. 3.Select items using the first fit heuristic until the knapsack cannot be filled any further. 34

35. Order Representation Approach Hinterding, R.:“Mapping, order-independent genes and the knapsack problem”, in Fogel, D., editor, Proc. of the First IEEE Conference on Evolutionary Computation, IEEE Press, Orlando, FL, pp. 13-17, 1994 For n item problem, a chromosome contains n genes and each gene represents an item with an identical integer number. The order of an item in the permutation representation can be viewed as the priority for it to be filled into the knapsack. Fitness function 35

36. Order Representation Example W=100 chromosome [ 1 6 4 7 3 2 5 ] the feasible solution [ 1 6 4 5 ] 36

37. Variable-length Representation Hinterding, R.:“Mapping, order-independent genes and the knapsack problem”, in Fogel, D., editor, Proc. of the First IEEE Conference on Evolutionary Computation, IEEE Press, Orlando, FL, pp. 13-17, 1994 The representation belongs to the direct encoding approach. The genes in the chromosome represent a legal knapsack because the total number of items in a legal knapsack is not fixed, the length of a chromosome is variable. The order of items in a chromosome has no meaning. g(x)=30≤W, f(x)=60 7 g(x)=80≤W, f(x)=120 7 2 g(x)=110>W, f(x)=130 gene # : g(x)=80≤W, f(x)=120 chromosome : 1 2 7 2 3 7 2 data set Sample of chromosome W=100 37

38. 1. Injecting a segment insert point parent 1 2 6 5 segment proto-child 2 5 7 6 5 parent 2 4 5 7 3 2. Deleting duplicated genes proto-child 2 5 7 6 3. Culling genes with first fit heuristic offspring 2 5 7 • Injecting crossover input: parent chromosomes output: offspring chromosomes step 1: Choose an insertion point in the first parent, and choose a segment in the second parent. step 2: Inject the segment into the first parent at the insertion point. step 3: Delete duplicate genes outside the segment to get a proto-child. step 4: Cull genes from the proto-child with the first fit heuristic to yield a legal knapsack. step 5: Output offspring chromosomes and stop. data set W=100 38

39. 1. Delete a selected number selected Selected point parent 2 6 5 proto-child 2 5 2. Append the items that were not present in the chromosome proto-child 1 3 4 2 5 7 6 3. Culling genes with first fit heuristic offspring 2 5 7 • Mutation procedure input: parent chromosome output: offspring chromosome step 1: Delete a number of genes at random. step 2: Append the items in random order that were not present in the chromosome. step 3: Cull genes from the proto-child with the first heuristic to yield a legal knapsack. step 4: Output offspring chromosome and stop. data set W=100 39

40. GP quick overview Developed: USA in the 1990’s Early names: J. Koza Typically applied to: machine learning tasks (prediction, classification…) Attributed features: competes with neural nets and alike needs huge populations (thousands) slow Special: non-linear chromosomes: trees, graphs mutation possible but not necessary (disputed!)

41. Tree based representation Trees are a universal form, e.g. consider Arithmetic formula Logical formula Program (x  true)  (( x  y )  (z  (x  y))) i =1; while (i < 20) { i = i +1 }

42. Tree based representation

43. Tree based representation (x  true)  (( x  y )  (z  (x  y)))

44. Tree based representation i =1; while (i < 20) { i = i +1 }

45. Parent 1 Parent 2 Child 1 Child 2

46. Intelligent Computing — Chapter 4 —Swarm Intelligence 46

47. Ant SystemStep Start Initialization To represent problem with route initialize pheromone State Transition Rule It satisfies the condition? Yes Pheromone Updating Rule No Every ant finds a route End Update the pheromone 47

48. Ant System While ( termination not satisfied ) create ants Find solutions Transition probability: Pheromone update Daemon activities {optional} Quantity of pheromone Heuristic distance α,β constants 48

49. Ant System While ( termination not satisfied ) create ants Find solutions Pheromone update Daemon activities {optional} Pheromone laid by each ant that uses edge (i,j) Evaporation rate 49

50. Ant System • Pheromone update : • Ant-Cycle Model • Ant-Quantity Model • Ant-Density Model • Ant-Cycle model uses the global information while Ant-Quantity and Density Model use the local information 50