“Why Design Information is Required to Find Improbable Complex Targets”

1. “Why Design Information is Required to Find Improbable Complex Targets” Robert J. Marks II Distinguished Professor Of Electrical and Computer Engineering RJMarksII@gmail.com EvoInfo.org

2. Abstract Engineers use models of science to improve quality of life.  Computational intelligence is such a useful engineering tool. It can create unexpected, insightful and clever results. Consequently, an image is often painted of computational intelligence as a free source of information. Although fast computers performing search do add information to a design, the needed information to solve even moderately sized problems is beyond the computational ability of the closed universe.  Assumptions concerning the solution must be included.  For targeted search, the requirement for added information is well known.   The need has been popularized in the last decade by the No Free Lunch theorems.  Using classic information theory, we show the added information for searches can, indeed, be measured.  The total information available prior to search is determined by application of Bernoulli's principle of insufficient reason. The added information measures the information provided by the evolutionary program towards achieving the available information.  Some recently proposed evolutionary models are shown, surprisingly, to offer negative added information to the design process and therefore perform worse than random sampling.

3. Menu • The Fossil Record • Computer Search • Evolutionary Search • Bernoulli’s Principle of Insufficient Reason • Blind Search • No Free Lunch Theorems • Active Information • Examples of Active Information • Final Thoughts

4. The Fossil Record • “In general, it is usually impossible or impracticable to test hypotheses about evolution in a particular species by the deliberate setting up of controlled experiments with living organisms of that species. We can attempt to partially to get around this difficulty by constructing [computer] models representing the evolutionary system we wish to study, and use these to test at least the theoretical validity of our ideas.” J. L. Crosby (~1965) • “The only way to see evolution in action is to make computer models” because “in real time these changes take aeons, and experiment is impossible.” Heinz Pagels (~1985) David Fogel, The Fossil Record, IEEE Press • “The Darwinian idea that evolution takes place by random hereditary changes and selection has from the beginning been handicapped by the fact that no proper test has been found to decide whether such evolution was possible and how it would develop under controlled conditions.” Nils Barricelli 1962

5. Menu • The Fossil Record • Computer Search • Evolutionary Search • Bernoulli’s Principle of Insufficient Reason • Blind Search • No Free Lunch Theorems • Active Information • Examples of Active Information • Final Thoughts

6. What is Computer Search?Searching for a Good Pancake Recipe How long do we cook on side one? How long do we cook on side two? Ten possibilities on each side: 100 combinations.

7. Searching for a Good Pancake Recipe Determining Fitness: Tim the Taster FITNESS: RANKS EACH PANCAKE ON A SCALE OF ONE TO TEN DESIGN CRITERION: 9 or above is an acceptable pancake.

8. Searching for a Good Pancake Recipe FITNESS LANDSCAPE: The plot of all 100 fitness values TARGET: 9 combinations meet the design criterion PROBABILITY OF SELECTION:p = 9/100 = 0.09 LANDSCAPE, TARGET and p ARE INITIALLY UNKNOWN

9. Searching for a Good Pancake Recipe ADD ONE MORE UNKNOWN: HEAT SETTING On THE STOVE Now there are 1000 possible combinations. THE CURSE OF DIMENSIONALITY!

10. Searching for a Good Pancake Recipe • Cooking the whole pancake • Pancake Mix: 1 to 10 cups • Eggs: 0 to 9 • Milk: 1 to 10 cups • Water: 0 to 9 cups 5. Salt: 1 to 10 pinches 6. Butter for Skillet: 0 to 9 pads 7. First Side Timing: 10 times 8. Second Side: 10 possibilities 9: Stove Setting: 10 Possibilities THE CURSE OF DIMENSIONALITY! Now there are ONE BILLION possible combinations.

11. Implicit Targets (Convergence) CRAFTED FITNESS LANDSCAPE: e.g. Revisit two parameter cooking example. Stove is electric and blows a fuse if either side of the pancake is cooked for more than a minute and half. THE CURSE OF DIMENSIONALITY! Now there are ONE BILLION possible combinations.

12. Searching for a Good Pancake Recipe Let a computer do it... THE CURSE OF DIMENSIONALITY!

13. Computer Search • Using software simulation, find a recipe that meets design criteria. • There are MANY search procedures

14. 100 Pancake Recipes Per Second

15. Defining “Impossible”

16. Menu • The Fossil Record • Computer Search • Evolutionary Search • Bernoulli’s Principle of Insufficient Reason • Blind Search • No Free Lunch Theorems • Active Information • Examples of Active Information • Final Thoughts

17. What is Evolutionary Search? • Simulation of Darwinian Evolution on a Computer How good is each solution? A set of possible solutions Computer Model Survival of the fittest Next generation Mutation Duplicate, Mutate & Crossover Keep a set of the best solutions

18. Yagi-Uda antenna (1954)  Space of all parameters. Parameters that give results better than Yagi-Uda T Designed by Evolutionary Search at NASA http://ic.arc.nasa.gov/projects/esg/research/antenna.htm Search in Engineering Design • Can we do better? Engineers… • Create a parameterized model • Establish a measure design’s fitness • Search the N-D parameter space

19. Random Search: You are told “Yes & No” (Success and no success) Target

20. Blind Search UHF http://www.youtube.com/watch?v=50uW0b7tWiM

21. Directed Search: Information is given to you... • e.g. • Warmer! • Steepest Descent • Conjugate Gradient Descent • Interval Halving Target

22. Menu • The Fossil Record • Computer Search • Evolutionary Search • Bernoulli’s Principle of Insufficient Reason • Blind Search • No Free Lunch Theorems • Active Information • Examples of Active Information • Final Thoughts

23. Bernoulli’s Principle of Insufficient Reason “...in the absence of any prior knowledge, we must assume that the events have equal probability” Jakob Bernoulli, ``Ars Conjectandi'' (``The Art of Conjecturing), (1713). e.g. A Lottery. If you have one of a million tickets, your chance of winning is the same as everyone else:

24. Bernoulli’s Principle of Insufficient Reason Laplace agreed: “[When] we have no reason to believe any particular case should happen in preference to any other” Arne Fisher, Charlotte Dickson, and William Bonynge, Mathematical Theory Of Probabilities & Its Applications To Frequency Curves & Statistical Methods, Macmillan (1922)

25. Menu • The Fossil Record • Computer Search • Evolutionary Search • Bernoulli’s Principle of Insufficient Reason • Blind Search • No Free Lunch Theorems • Active Information • Examples of Active Information • Final Thoughts

26. Blind Searches... 27 keys Apply Bernoulli's principle of insufficient reason “in the absence of any prior knowledge, we must assume that the events have equal probability Jakob Bernoulli, ``Ars Conjectandi'' (``The Art of Conjecturing), (1713). • Monkeys at a typewriter… Information Theoretic Equivalent: Maximum Entropy (A Good Optimization Assumption)

27. How Does Moore’s Law Help in Blind Search? Computer today searches for a target of B =10000 bits in a year. Double the speed. Faster Computer searches for a target of B + 1 =10001 bits in a year.

28. Converting Mass to Computing Power • Minimum energy for an irreversible bit Von Neumann-Landaurer limit = ln(2) k T = 1.15 x 10 -23joules • Mass of Universe ~ 1053 kg. Convert all the mass in the universe to energy (E=mc2) , we could generate 7.83 x 1092 Bits 1. Assuming background radiation of 2.76 degrees Kelvin

29. Define Impossible: Anything requiring 1093 bits or more. Seth Lloyd in Physical Review, says 10120 Seth Lloyd, “Computational Capacity of the Universe”, Physical Review Letters 88(23) (2002): 7901-7904.

30. L How Long a Phrase? Target • IN THE BEGINNING ... EARTH • JFD SDKA ASS SA ... KSLLS • KASFSDA SASSF A ... JDASF • J ASDFASD ASDFD ... ASFDG • JASKLF SADFAS D ... ASSDF . . . • IN THE BEGINNING ... EARTH Expected number = NL 4.7549 bits per letter for 26 letters and a space

31. How Long a Phrase from the Universe? For N = 27, p=N-L L = 63 characters Number of bits expected for a random search p=N-L 7.83 x 1092bits = NL log2NL

32. Using Lloyd’s 10120 bits for 101000 universes gives 836 letter search. How Long a Phrase from the Multiverse?

33. Does Quantum Computing Help? Quantum computing reduces search time by a square root.  L. K. Grover, “A fast quantum mechanical algorithm for data search”, Proc. ACM Symp. Theory Computing, 1996, pp. 212--219.

34. Pr[tT ] =  Probability Search Space Pr()=1 Target T t Targeted Search

35. Acceptable solutions  T Fitness Each point in the parameter space has a fitness. The problem of the search is finding a good enough fitness.

36. Search Algorithms Steepest Ascent Exhaustive Newton-Rapheson Levenberg-Marquardt Tabu Search Simulated Annealing Particle Swarm Search Evolutionary Approaches Problem: In order to work better than average, each algorithm implicitly assumes something about the search space and/or location of the target.

37. Menu • The Fossil Record • Computer Search • Evolutionary Search • Bernoulli’s Principle of Insufficient Reason • Blind Search • No Free Lunch Theorems • Active Information • Examples of Active Information • Final Thoughts

38. No Free Lunch Theorem With no knowledge of where the target is at and no knowledge about the fitness surface, one search performs, on average, as good as any another.

39. No Free Lunch Theorem Made EZ Find the value of x that maximimizes the fitness, y. y x Nothing is known about the fitness, y.

40. Another illustration...52 Cards.Find the Ace of Spades

41. Cullen Schaffer (1994)A Conservation Law for Generalization Performance “About half of the people in the audience to which my work was directed told me that my result was completely obvious and common knowledge–which is perfectly fair. Of course, the other half argued just as strongly that the result wasn’t true.” (Private Correspondence)

42. Quotes on the need for added information for targeted search … • “…unless you can make prior assumptions about the ... [problems] you are working on, then no search strategy, no matter how sophisticated, can be expected to perform better than any other” Yu-Chi Ho and D.L. Pepyne, (2001). • No free lunch theorems “indicate the importance of incorporating problem-specific knowledge into the behavior of the [optimization or search] algorithm.” David Wolpert & William G. Macready (1997). ``Simple explanantion of the No Free Lunch Theorem", Proceedings of the 40th IEEE Conference on Decision and Control, Orlando, Florida, "No free lunch theorems for optimization", IEEE Trans. Evolutionary Computation 1(1): 67-82 (1997).

43. Therefore... • Nothing works better, on the average, than random search. • For a search algorithm like evolutionary search to work, we require active information.

44. Evolutionary Search... • Evolutionary search is “able to adapt solutions to new problems and do not rely on explicit human knowledge.” David Fogel* BUT, the dominoes of an evolutionary program must be set up before the are knocked down. Recent results (NFL) dictate there must be implicitlyadded information in the crafting of an evolution program. * (emphasis added D. Fogel, Review of “Computational Intelligence: Imitating Life,” IEEE Trans. on Neural Networks, vol. 6, pp.1562-1565, 1995.

45. Evolutionary Computing e.g. setting up a search requires formulation of a “fitness function” or a “penalty function.” Michael Healy, an early pioneer in applied search algorithms, called himself a “penalty function artist.”

46. 9 6 ? Can a computer program generate more information than it is given If a search algorithm does not obey the NFL theorem, it “is like a perpetual motion machine - conservation of generalization performance precludes it.” Cullen Schaffer (1994) • Cullen Schaffer, 1994. “A conservation law for generalization performance,”in Proc. Eleventh International Conference on Machine Learning, H. Willian and W. Cohen, San Francisco: Morgan Kaufmann, pp.295-265.

47. Conservation of Information A computer can create information no more than an iPod can create music

48. Menu • The Fossil Record • Computer Search • Evolutionary Search • Bernoulli’s Principle of Insufficient Reason • Blind Search • No Free Lunch Theorems • Active Information • Examples of Active Information • Final Thoughts

49. Conservation of Information “The [computing] machine does not create any new information, but it performs a very valuable transformation of known information.” Leon Brillouin, Science and Information Theory (Academic Press, New York, 1956).

50. Shannon Information Axioms • Small probability events should have more information than large probabilities. • “the nice person” (common words  lower info) • “philanthropist” (less used  more information) • Information from two disjoint events should add • “engineer”  Information I1 • “stuttering”  Information I2 • “stuttering engineer”  Information I1 + I2