Notes on Backpropagation

1. Notes on Backpropagation Alex Churchill

2. Feed Forward • Node C = sigmoid(A * weightca + B * weightcb) C D

3. Feed Forward • Node C = sigmoid(0.1 * 0.1+ 0.7*0.5) C D

4. Feed Forward • Node C = sigmoid(0.01+0.35) = 0.59 C 0.59 D

5. Feed Forward • Node D = sigmoid(A * weightda + B * weightdb) C 0.59 D

6. Feed Forward • Node D = sigmoid(0.1 * 0.3+ 0.7*0.2) C 0.59 D

7. Feed Forward • Node D = sigmoid(0.03+0.14) = 0.54 C 0.59 E D 0.54

8. Feed Forward • Node E = sigmoid(C * weightec + D * weighted) C 0.59 E D 0.54

9. Feed Forward • Node E = sigmoid(0.59*0.2 + 0.54*0.1)=0.542 C 0.59 E D 0.54

10. Feed Forward • Node E = sigmoid(0.59*0.2 + 0.54*0.1)=0.542 C 0.59 E 0.542 0.542 D 0.54

11. Backpropagation • Calculate error for each output neuron at the output layer (L) • For each hidden layer (L-1 to L – n) pass the error backwards from the layer above • Update the weights connecting the last hidden layer (L-1) to the output layer (L) • Update the weights connecting each lower layer

12. Backpropagation • Calculate error (δk)for each output neuron (k) at the output layer (L) This is calculated using: δk=(yk-tk)*g'(xk) Where g’ is the first derivative of the sigmoid and xk is the pre-sigmoided output

13. Backpropagation δk=(yk-tk)*g'(xk) δk=(0.542-1)*g'(xk)=(0.542-1)*(0.542)*(1-0.542) = -0.114 Target = 1 C Learning rate = 1 0.59 E 0.542 0.542 D 0.54

14. Backpropagation δk=(yk-tk)*g'(xk) δk=(0.542-1)*g'(xk)=(0.542-1)*(0.542)*(1-0.542) = -0.114 Target = 1 C Learning rate = 1 δk=-0.114 0.59 E 0.542 0.542 D 0.54

15. Backpropagation 2. For each hidden layer (L-1 to L – n) pass the error backwards from the layer above This is calculated using: Where j is the hidden neuron and k is the output neuron

16. Backpropagation δc=(wecδk)*g'(xc) = 0.2 * -0.114 * 0.59*(1-0.59)=-0.0055 δd=(wedδk)*g'(xd) = 0.1 * -0.114 * 0.54*(1-0.54)=-0.0028 Target = 1 C Learning rate = 1 δk=-0.114 0.59 E 0.542 0.542 D 0.54

17. Backpropagation δc=-0.0055 δc=(wecδk)*g'(xc) = 0.2 * -0.114 * 0.59*(1-0.59)=-0.0055 δd=(wedδk)*g'(xd) = 0.1 * -0.114 * 0.54*(1-0.54)=-0.0028 Target = 1 C Learning rate = 1 δk=-0.114 0.59 E δd=-0.0028 0.542 0.542 D 0.54

18. Backpropagation 3. Update the weights connecting the last hidden layer (L-1) to the output layer (L) This is calculated using: Where j is the hidden neuron and k is the output neuron. aj is the sigmoided output of the hidden neuron

19. Backpropagation δc=-0.0055 Wec=wec-ηδeC = 0.2 - 1* -0.114*0.59=0.267 Wed=wed -ηδed = 0.1 - 1* -0.114*0.54=0.162 Target = 1 C Learning rate = 1 δe=-0.114 0.59 E δd=-0.0028 0.542 0.542 D 0.54

20. Backpropagation δc=-0.0055 Wec=wec-ηδeC = 0.2 - 1* -0.114*0.59=0.267 Wed=wed -ηδed = 0.1 - 1* -0.114*0.54=0.162 Target = 1 C Learning rate = 1 0.267 0.59 E δd=-0.0028 0.542 0.542 D 0.162 0.54

21. Backpropagation 4. Update the weights connecting each lower layer. This is calculated using: Where j is the hidden neuron (or input neuron) in the layer below and k is the hidden neuron in the layer above.

22. Backpropagation δc=-0.0055 Wca=wca-ηδcA = 0.1 - 1* -0.0055*0.1=0.1005 Wcd=wcd-ηδdA = 0.3 - 1* -0.0028*0.1=0.3003 Target = 1 C Learning rate = 1 0.267 0.59 E δd=-0.0028 0.542 0.542 D 0.162 0.54

23. Backpropagation δc=-0.0055 Wca=wca-ηδcA = 0.1 - 1* -0.0055*0.1=0.1005 Wcd=wcd-ηδdA = 0.3 - 1* -0.0028*0.1=0.3003 Target = 1 C 0.1005 Learning rate = 1 0.267 0.59 0.3003 E δd=-0.0028 0.542 0.542 D 0.162 0.54

24. Feed forward

25. Iris

26. Iris