1 / 48

Visualization of hidden node activity in a feed forward neural network

Visualization of hidden node activity in a feed forward neural network. Adam Arvay. Feed forward neural networks. Function finding device Learns a function to transform a set of inputs into the desired output Uses supervised learning. Network building software. PyBrain v0.3

lecea
Télécharger la présentation

Visualization of hidden node activity in a feed forward neural network

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Visualization of hidden node activity in a feed forward neural network Adam Arvay

  2. Feed forward neural networks • Function finding device • Learns a function to transform a set of inputs into the desired output • Uses supervised learning

  3. Network building software • PyBrain v0.3 • Modular machine learning library for Python • PyBrain is short for Python-Based Reinforcement Learning, Artificial Intelligence and Neural Network Library

  4. Visualization tools • NetworkX • Used for keeping track of node names and edges • matplotlib/pyplot/pylab • Drawing everything

  5. Data set • Iris data set • 150 total data points • 4 inputs • 3 outputs (classifications) • 50 of each classification type • CSV file

  6. Networks analyzed • 3 networks were constructed with different numbers of hidden layers • 4 input nodes (linear) • 4, 7, 10 hidden nodes (sigmoid) • 3 output nodes (softmax) • Trained with back-propigation • Training/validation data selected randomly • 250 epochs

  7. Visualizations • Mean squared errors during training • Network state • Average activation levels • Absolute hidden node sensitivity • Weighted hidden node sensitivity • Activation scatter

  8. Mean squared error • Quick way to evaluate training efficacy • Plot the error vs. training time (epochs) • Expect error to go down with increased training • Greatly depends on quality of training data

  9. Mean squared error

  10. Network state visualization • Displays abstract logical connections between nodes in a spatial layout • Size to represent activation level • Colored and line style used to depict connection type. Black for positive, red dashed for negative

  11. Network state visualization • A snapshot of what the network is currently doing • Interactivity: • Shows the state of the network under a particular activation • Visible edge threshold magnitude can be set • Edges can be labeled

  12. Network state

  13. Network state

  14. Network state

  15. Network state all connections

  16. Network state all connections with all labels

  17. Network state 7 nodes no labels

  18. Network state 7 nodes

  19. Network state 7 nodes

  20. Network state 10 nodes

  21. Network state 10 nodes

  22. Network state 10 all connections

  23. Network state • Gives information about current state of network • Interactive • Can get cluttered with many nodes and connections • Difficult to see trends

  24. Average activation levels • Gives an idea of the network behavior over time for a particular classification type • Can detect pattern differences in hidden layer between classification types • Shows the average activation level of a hidden node across a classification type • No interactivity

  25. 4 nodes, setosa

  26. 4 nodes, versacolor

  27. 4 nodes, virginica

  28. 7 nodes setosa

  29. 7 nodes versacolor

  30. 7 nodes virginica

  31. 10 nodes setosa

  32. 10 nodes versacolor

  33. 10 nodes virginica

  34. Average activation • Can see some patterns between classificaitons • Easy to spot changes and non-changes • Doesn’t depict the variance in the activations

  35. Absolute hidden node sensitivity • A quick way to determine the sensitivity of a hidden node to its inputs • Can detect nodes which are insensitive to all inputs • Can detect which inputs are ignored by all nodes • Can detect patterns of connections across nodes

  36. Hidden node sensitivity

  37. Hidden node sensitivity

  38. Hidden node sensitivity

  39. Weighted sensitivity • Accounts for differences in magnitude of the input parameters • In the iris data set, the first input has a much larger average value than the last input. • Normalizes the weights to the inputs

  40. Weighted sensitivity

  41. Weighted sensitivity

  42. Weighted sensitivity

  43. Comparisonnon-weighted vs weighted

  44. Activation scatter • Used along with average activation to get more information about the activation activity of hidden nodes across a classification type • Can get a sense of the variance of a particular node • Color used to represent a node along with data labels.

  45. Activation scatter setosa

  46. Activation scatter versacolor

  47. Activation scatter virginica

  48. Conclusion • 4 main visualization tools • Training data • Network state • Average activation • Hidden node sensitivity • Designed to be used with 3 layer networks with arbitrary number of nodes per layer

More Related