Computing in Complex Systems

1. Center for Engineering Science Advanced Research OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY Computing in Complex Systems J. Barhen Computing and Computational Sciences Directorate Research Alliance for MinoritiesFall WorkshopORNL Research Office BuildingDecember 2, 2003

2. Advanced Computing Activities at CESAR In 1983 DOE established CESAR at ORNL. Its purpose was to conduct fundamental theoretical, experimental, and computational research in intelligent systems. Over the past decade, the Center has experienced tremendous growth. Today, its primary activities are in support of DOD and the Intelligence Community. Typical examples include: • missile defense: BMC3, war games, HALO-2 project, multi-sensor fusion • sensitivity and uncertainty analysis of large computational models • laser array synchronization (directed energy weapons) • complex systems: neural networks, global optimization, chaos • quantum optics applied to cryptography • mobile cooperating robots, multi-sensor and computer networks • nanoscale science (friction at the nanoscale, interferometric nanolithography) Within the CCS Directorate, revolutionary computing technologies (optical, quantum, nanoscale, neuromorphic) are an essential focus of CESAR’s research portfolio. CESAR sponsorsinclude:MDA, DARPA, Army, OSD/JTO, NRO, ONR, NASA, NSA, ARDA, DOE/SC, NSF, DOE/FE, and private industry.

3. The Global Optimization ProblemIllustrative Example of Computing in Complex Systems Nonlinear Optimization problems arise in every field of scientific, technologic, economic, or social interest. Typically, • The objective function (the function to be optimized) is multimodal, i.e., it possesses many local minima in the parameter region of interest • In most cases it is desired to find the local minimum at which the function takes its lowest value, i.e., the global minimum The design of algorithms that can reach and distinguish between local and global minima is known as the global optimization problem. Examples abound: • Computer Science: design of VLSI circuits, load balancing, … • Biology: protein folding • Geophysics: determination of unknown geologic parameters from surface measurements • Physics: elasticity, hydrodynamics, … • Industrial technology: optimal control, design, production flow, … • Economics: transportation, cartels, …

4. Problem Formulation Definitions • x is a vector of state variables or parameters • f is referred to as the objective function Goal Find the values fGandxGsuch that •  is the domain of interest over which one seeks the global minimum. It is assumed to be compact and connected. • without loss of generality, we will take  as the hyper parallelepiped

5. Local vs Global Minima

6. Why is Global Optimization so Difficult?Illustration of Practical Challenges • Complex Landscapes • we need to find global minimum of functions • of manyvariables • Typical problem size is • (102 – 105) variables • Difficulty • number of local minima grows • exponentially with the • number of variables • local and global minima have • the same signature, namely • zero gradient Schubert function: This function arises in signal processing applications. It is used as one of the SIAM benchmarks for Global Optimization. Even its two dimensional instantiation exhibits a complex landscape.

7. Leading Edge Global Optimization Methods The Center for Engineering Science Advanced Research (CESAR) at the Oak Ridge National Laboratory (ORNL) has been developing, demonstrating, and documenting in the open literature leading edge global optimization (GO) algorithms. What is the Approach? • three complementary methods address GO challenge • exploit different aspects of problem but can be used in synergistic fashion What are the Options? • TRUST: fastest published algorithm for searching complex landscapes via tunneling • NOGA: performs nonlinear optimization while incorporating uncertainties from model and from external information (sensors, …) • EO: exploits the availability of information typically available to the user but never exploited by conventional optimization tools Goal: Further develop, adapt, and demonstrate these methods on relevant DOE, DOD, and NASA applications where major impact is expected.

8. Leading Edge Global Optimization MethodsTRUST What is TRUST ? • a new, extremely powerful global optimization paradigm developed at CESAR / ORNL How does it work ?three innovative concepts • subenergy tunneling: a nonlinear transformation that creates a virtual landscape where all function values greater than the last found minimum are suppressed • non-Lipschitzian “terminal” repellers: enable escape from local minima by “pushing” the solution flow under the virtual landscape • stochastic Pijavskyi cones: eliminate unproductive regions by using information on the Lipschitz constant of the objective function acquired during the optimization process • iterative decomposition & recombination of large scale problems How does it perform ? • unprecedented speedand accuracy: overall efficiency up to 3 orders of magnitude higher than best publicly available competitors for SIAM benchmarks • successfully tested on large-scale seismic imaging problem • outstanding performance led to article inScience (1997), to R&D 100 award (1998), and to a patent in 2001.

9. TRUSTTerminal Repeller Unconstrained Subenergy Tunneling

10. TRUSTComputational Approach

11. We seek global minimum of blue function Current local minimum Effective tunneling Subenergy tunneling transformation is applied to shifted (green) function Motion on virtual surface Uniqueness of TRUST • Virtual objective function E( x, x* ) is a superposition of two contributing terms • Esub (x, x*): subenergy tunneling • Erep (x, x*): repelling from latest found local minimum • Its effect is to transform the current local minimum of f(x) into a global maximum, while preserving any lower laying local minima Key Advantage of TRUST • Gradient descent applied to f(x) and initialized at x*+can not escape from the basin of attraction of x* • Gradient descent applied to E( x, x* ) and initialized at x*+always escapes it. • TRUST has a • global descent property. Erep Esub

12. Leading Edge Global Optimization Methods • Comparison of TRUST performance to leading publicly available competitors for SIAM benchmarks • data correspond to number of function evaluations needed to reach global minimum • symbol  indicates that no solution was found for method under consideration • benchmark functions: BR (Branin), CA (camelback), GP (Goldstein-Price), RA (Rastrigin), SH (Shubert), • H3 (Hartman) • methods: SDE (stochastic differential equations), GA/SA (genetic algorithms and simulated annealing), • IA (interval arithmetic), Levy TUN (conventional Levy tunneling), Tabu (Tabu search)

13. Leading Edge Global Optimization Methods NOGA • The explicit incorporation of uncertainties into the optimization process is essential for the design of robust mission architectures and systems • NOGA = method for Nonlinear Optimization and Generalized Adjustments • explicitly computes the uncertainties in model predicted results in terms of uncertainties in intrinsic model parameters and inputs • determine best-estimates of model parameters and reduces uncertainties by consistently incorporating external information • NOGA methodology is based on the concepts and tools of sensitivity and uncertainty analysis. It performs a non-linear optimization of a constrained Lagrange function that uses the inverse of a generalized total covariance matrix as natural metric EO • EO = Ensemble Optimization • Builds on systematic study on the role that additional information may have in significantly reducing the complexity of the GOP  • while in most practical problems additional information is readily available either at no cost at all or at rather low cost, present optimization algorithms cannot take advantage of it to increase the efficiency of the search.  • to overcome this shortcoming, we have developed EO, a radically new class of optimization algorithms that can readily fold in additional information and - as a result – dramatically increase their efficiency

14. Leading Edge Global Optimization MethodsSelected References TRUST • Barhen, J., V. Protopopescu and D. Reister, “TRUST: A Deterministic Algorithm for Global Optimization”, Science, 276, 1094-1097(1997). • Reister, D., E. Oblow, J. Barhen, and J. DuBose, “Global Optimization to Maximize Stack Energy”, Geophysics, 66(1), 320-326 (2001). NOGA • Barhen, J. and D. Reister, “Uncertainty Analysis based on Sensitivities Generated using Automated Differentiation”, Lecture Notes in Computer Science, 2668, 70-77, Springer (2003). • Barhen, J., V. Protopopescu, and D. Reister, “Consistent Uncertainty Reduction in Modeling nonlinear Systems”, SIAM Journal of Scientific Computing (in press, 2003). EO • Protopopescu, V. and J. Barhen, "Solving a Class of Continuous Global Optimization Problems using Quantum Algorithms", Physics Letters, A 296, 9-14 (2002). • Protopopescu, V., C. d’Helon, and J. Barhen, “Constant-time Solution to the Global Optimization Problem using Bruschweiler’s Ensemble Search Algorithm, Jour. Phys., A 36(24), L399-L407 (2003).

15. Frontiers in Computing • Three decades ago, fast computational units were only present in vector super-computers. • Twenty years ago, the first message-passing machines (Ncube, Intel) were introduced. • Today, the availability of fast, low-cost chips, has revolutionized the way calculations are performed in various fields, from personal workstation to tera-scale machines. • An innovative approach to high performance, massively parallel computingremains a key factor for progress in science and national defense applications. • In contrast to conventional approaches, one must develop computationalparadigms that exploit, from the onset (1)the concept of massive parallelismand(2)the physics of the implementation device. • Ten to twenty years from now, asynchronous,optical, nanoelectronic, biologically inspired, and quantum technologies have the potential of further revolutionizing computational scienceand engineering by • offering unprecedented computational power for a wide class of demanding applications • enabling the implementation of novel paradigms