1 / 21

CS623: Introduction to Computing with Neural Nets (lecture-13)

This lecture covers the fundamental concepts of Hopfield Networks, focusing on their energy minimization capabilities. We analyze weight configurations and energy states, illustrating how networks stabilize in O(n²) steps. Special attention is given to optimization problems, using examples like the Traveling Salesman Problem and Job Scheduling. The architecture and neuron arrangements are discussed, emphasizing the constraints necessary for effective problem-solving. Finally, we establish a connection between energy minimization and combinatorial optimization, reinforcing the relevance of Hopfield Networks in practical applications.

tyrell
Télécharger la présentation

CS623: Introduction to Computing with Neural Nets (lecture-13)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CS623: Introduction to Computing with Neural Nets(lecture-13) Pushpak Bhattacharyya Computer Science and Engineering Department IIT Bombay

  2. Hopfield Net: an o(n2) algorithm • Consider the energy expression • E has [n(n-1)]/2 terms • Nature of each term • wij is a real number • xi and xj are each +1 or -1

  3. Ehigh Egap Elow No. of steps taken to reach stability • Egap = Ehigh - Elow

  4. Analysis of the weights and consequent Ehigh and Elow • Wilis any weight with upper and lower bounds as Wmax and Wminrespectively. Suppose wmin≤ wij ≤ wmax wmax > wmin • Case 1: wmax > 0, wmin > 0 Ehigh = (1/2) * wmax * n * (n-1) Elow = -(1/2) * wmax * n * (n-1)

  5. Continuation of analysis of Ehighand Elow • Case 2: wmin < 0, wmax < 0 Ehigh = (1/2) * wmin * n * (n-1) Elow = -(1/2) * wmin * n * (n-1) • Case 3: wmax > 0, wmin < 0 Ehigh = (1/2) * max(|wmax|,|wmin|) * n*(n-1) Elow = -(1/2) * max(|wmax|,|wmin|) * n*(n-1)

  6. The energy gap • In general, Ehigh Egap = max(|wmax|, |wmin|) * n*(n-1) Elow

  7. To find ΔEmin ΔEp = (xpinitial - xpfinal) * netp where ΔEp is the change in energy due to the pth neuron changing activation. | ΔEp | = | (xpinitial - xpfinal) * netp| = 2 * | netp| where netp = Σnj=1, j≠pwpjxj

  8. To find ΔEmin • [Σnj=1, j≠pwpjxj]min is determined by the precision of the machine computing it. • For example, assuming 7 places after the decimal point, the value cannot be lower than 0.0000001 [it can be 0, but that is not of concern to us, since the net will continue in the same state] • Thus ΔEmin is a constant independent of n, determined by the precision of the machine.

  9. Final observation: o(n2) • It is possible to reach the minimum independent of n. • Hence in the worst case, the number of steps taken to cover the energy gap is less than or equal to [max(|wmax|,|wmin|) * n * (n-1)] / constant • Thus stability has to be attained in O(n2) steps

  10. Hopfield Net for Optimization • Optimization problem • Maximizes or minimizes a quantity • Hopfield net used for optimization • Hopfield net and Traveling Salesman Problem • Hopfield net and Job Scheduling Problem

  11. The essential idea of the correspondence • In optimization problems, we have to minimize a quantity. • Hopfield net minimizes the energy • THIS IS THE CORRESPONDENCE

  12. dCD D C dDA dBC dAC dBD A B dAB Hopfield net and Traveling Salesman problem • We consider the problem for n=4 cities • In the given figure, nodes represent cities and edges represent the paths between the cities with associated distance.

  13. Traveling Salesman Problem • Goal • Come back to the city A, visiting j = 2 to n (n is number of cities) exactly once and minimize the total distance. • To solve by Hopfield net we need to decide the architecture: • How many neurons? • What are the weights?

  14. Constraints decide the parameters • For n cities and n positions, establish city to position correspondence, i.e. Number of neurons = n cities * n positions • Each position can take one and only one city • Each city can be in exactly one position • Total distance should be minimum

  15. Architecture pos(α) • n * n matrix where rows denote cities and columns denote positions • cell(i, j) = 1 if and only if ith city is in jth position • Each cell is a neuron • n2 neurons, O(n4) connections city(i)

  16. Expressions corresponding to constraints • Each city in one and only one position i.e. a row has a single 1. • Above equation partially ensures each row has a single 1 • xiα is I/O cost at the cell(i, α)

  17. Expressions corresponding to constraints (contd.) 2. Each position has a single city i.e. each column has at most single 1.

  18. Expressions corresponding to constraints (contd.) 3. All cities MUST be visited once and only once

  19. Expressions corresponding to constraints (contd.) • E1, E2, E3 ensure that each row has exactly one 1 and each column has exactly one 1 • If we minimize E1 + E2 + E3 • Ensures a Hamiltonian circuit on the city graph which is an NP-complete problem.

  20. Constraint of distance 4. The distance traversed should be minimum dij = distance between city i and city j

  21. Expressions corresponding to constraints (contd.) • We equate constraint energy: EProblem = Enetwork (*) Where, Eproblem= E1+E2+E3+E4 and Enetwork is the well known energy expression for the Hopfield net • Find the weights from (*).

More Related