Neural Network Similarity Joel Lehman
Outline • Genetic Algorithms • Neural Networks • Evolving Neural Networks • Neural Network Similarity Problem • NP-Completeness Proof
Genetic Algorithms • Inspired by the apparent power of Natural Evolution • Usually applied to optimization problems • Consists of: • A population of candidate solutions • A mechanism for generating variation • A measure of goodness used for selection
Neural Networks • Computational abstraction of biological brains • Consist of nodes (neurons) and connections (synapses) • Can be drawn as a weighted digraph
Evolving Neural Networks • Intelligence on earth evolved in form of biological brains • Using GA+NN is a very rough abstraction of how intelligence evolved • Can be used in many tasks • Robot control, game playing, classification, etc. • How to define genetic operators that operate on NN?
Evolving Neural Networks • Use a ‘direct encoding’ • Represent NN directly as a graph • Mutation is easy • Simply add new connections, nodes, or change connection weights • Crossover is hard • How to recognize what parts of one NN correspond to parts of another • Without this idea of ‘homology’ naïve crossover will likely just destroy the functionality of both NN
Neural Network Similarity Problem • Basically, question is: what parts of one NN are relatively the same to parts of a second NN • If we know this, we can intelligently do crossover to create meaningful offspring NN
Neural Network Similarity Problem • A Neural Network is composed of a 4-tuple: (Input Nodes, Output Nodes, Hidden Nodes, Connections) • Two NNs are ‘compatible’ if they share same Input and Output Nodes • A subnetwork of a neural network A is a NN compatible with A, with hidden nodes and connections that are subsets of those in A
Neural Network Similarity Problem • Given two compatible NNs, N1 and N2, an integer k, and a real number l • Does there exist a subnetwork NS1 of N1 with at least k connections isomorphic (disregarding connection weights) to a subnetwork NS2 of N2 such that the summed difference in connection weights between NS1 and NS2 are less than l.
NP Completeness Proof • First we must show it is in NP. • A witness can give the partial map between hidden nodes and connections in N1 to N2. It can then be verified in polynomial time that the conditions of k and l are satisfied
NP Completeness Proof • Basically this is an augmented version of subgraph isomorphism • Subgraph isomorphism • Given two graphs G1, G2, does there exist a subgraph of G1 with at least k edges that is isomorphic to a subgraph of G2?
NP Completeness Proof • Easy to map an instance of SGI into NNSP: • Create a NN from each graph by the following: • Set input and output nodes to null set • Vertices in the graph translate directly to hidden nodes • Edges in the graph translate directly to connections in the NN all with identical weights • K in NNSP is set to k from SGI • L in NNSP is set to 0 • Intuitively we know this works • We’ve stripped away all the augmentations
NP Completeness Proof • Given yes instance of SGI • The mapped instance of NNSP will also have an subgraph isomorphism, the NNs are isomorphic to the given graphs, and the L criterion is satisfied • Given yes instance of NNSP • We know there are no hidden/input nodes • All weights are 1.0 (and thus insignificant to L criterion) • The isomorphism between NN/Graph and the decision question means there must be a SGI of size k