Introduction to Computational Methods: Approximations in Scientific Computing

CSE 551 Computational Methods 2018/2019 Fall Chapter 1 Introduction

Outline Introduction Computational Problems Approximations in Scientific Computations Broad Classification of Computational methods Preliminaries – Nested Multiplication Preliminaries – Taylor Series Course Related Issues

References • Based on • M. TG. Heath, Scientific Computing: An Introduction, 2ed ed, Mc Graw Hill. • Chapter 1: Introduction • S. . Chapra, Numerical Methods for Engineers: with Software and Programming Applications, Mc Craw Hill. • Introduction to Pert I

Introduction • numerical analysis – scienfific computing • design and analysis of algorithms • for solving mathematical problems with aritmetic operations • in many fields – • science and engineering • recently social sciences • quantities continuous v.s. discrete • functions and equations – underlying variables • time, distance, velocity, temperature, presure, density,stress and like

most problems in continuous math • derivatives, integration, nonlinearities • cannot be solved exactly – in finite number of steps • iterative process – converges to a solution • the answer is approximately correct • close enough to the desired result

finding rapidly convergent iterative algorithms • assesing accuricy of rssulting approximation • if rapid • some problems with finite algorithms – systems of linear eq. – better with iterative methods

Approximations • effects of approxmimations • many solution teckhniques • approximations – of verious types • even the aritmetic • digital computer cannot represent all real numbers exactly • numerical algorithms • efficient, reliable and accurate

NonComputer Methods • analytical or exact methods: • limited class of problems • linear models • simple geometries • low dimensionality • useful and excelent inside to the behavior of systems • limited practical value • most real problems • nonlinearities • complex shapes and processes

Graphical solutions: • characterize behavior • plots or chats • complex problems but results are not very precise • low dimensional – three or fewer • e.g., phase diagrams in thermodynamics • much effort and energy on solution technique • rather than problem formuolation and interpretation

Computational Problems • many problems scientific computing from • science or engineering • social sciences, business – computational social scinece • ultimate aim • understand some natural, social phenomena • design a device • computational simulation: • representation or emulation of a physical, social system or a process using computers • greatly enhence scientific understanding by allowing the investigation of situations • difficult or impossible – ttheoretical, observational or experimental means alone

Examples • In astrophysics – behavior of two collding black hodes • too complicated to determine theoretically – analytical methods • impossible to observe directly • dublicate in lab • to simulate it computationally requires only • an approximate mathematical representation – Einstein’s equations of general relativity • an algorithm to solve these equations numerically • a sufficiently powerful computer

Examples (cont.) • investigate normal situations with less cost and time • Engineering design – large number of design options are tried • quickly, inexpensively, safely • than with treditional “bulid-and-test” methods usingf physical prototypes v.s. virtual prototyping • e.g., improving automobile safety – crash testing • less expensive and dangerous on a computer • space of design parameter explored more throughly • drug design – computational biochemistry • social policy programms – impossible to meke experiments on society

Problem Solving Process in Computational Simulation • Develop a methematical model • expressed with some equations some type – equation based modeling EBM v.s. agent-based-modeling ABM • representing the physical phenomena or the system • Develop algorthms to solve the equations numerically • Implement the algorithms on a computer • Run the algorithms on the computer • Reprsent the computed results – comprehensible form – graphical visuliztion • Interpret and validate the computed results

Step 1 – mathematical modeling • domain knowldge particular scientific or engineering disiplines • applied mathematics • Step 2,3 – designing, analysing , implementging numercal algorthms – main subject of scientific computing • Principles and methods of scientific computing • studied fairly broad level in generality

but keep in mind • specific sources of a problem and the uses • original problem formulation may affect • accracy of numerical results which affects • interpretation and validation of these reslults

Well-posedness v.s. Ill-posssedness • a mathematical problem is well-posed • if a solution exists unique and • depends continuously on the problem data • a small change in data does not cause an abrubt disproportionate change in the solution • in numerical computations • such perturbations are usually inavitable • well-possedness - highly desirable • but not always atchievable

An Example • e.g., infering the internal structure of a system from external observations • in tomography or seismology – mathematical problems – ill-posed • distincly different internal configurations • may have indistinguishable external apperances

Sensitivity • even a problem is well-possed • the solution may be sensitive to perturbations to perturbations in data or parameters • develop quantitative measures of sensitivity • local and global sensitivity • robustness to alternative assumptions or processes • Sensitivity of algorthms stable algorithms

General Strategy • replace a dificult problem with an easier one • same or closly related solutin • E.g.: • infinite dimensional spaces with finite dimensional spaces • infinite processes with finite processes • integrals or infinite series with finite sums • derivatives with finite differences • differential equations with difference equations (algebric equations) • nonlinear problems with linear problems

Replacements (cont.) • high order systems with low order systems • complicated functions with simple functions • polynomials • general matrices with matrices with a simpler form

Example • to solve a system of nunlinear differential equations • first, replace with system of nonlinear algebric equations – difference equations • then, replace the nonlinear system with a linear one • then, replace the natrix of the linear system with a special form • solution is easy to compute • at each step – verify that • within some tolerance of the true solution

an alternative problem(s) easier to solve • a transformation of the given problem to the alternative one • preserves the solution in some sense • much effort • identify class of problems with simple solutions • solution preserving transformations into these classes

ideally – solution of transformed problem is identical to the original problem • not always possible – approximate • accuracy arbitarily good – additional work and storage • primary concern • estimating accuracy of such an approximate solution • establishing convergence to the true solution in the limit

Approximations in Scientific Computing • Sources of approximation • some before the computation begins • Modeling: • some features of the system under study may be ommited or simplified • friction, viscosity, air resistance • Emprical measurments: • lab instruments – finite precision • accuricy – further limited • small sample size • reading - random noice or systematic bias

e.g., • even most careful measurments of physical constants – Newton’s gravitgational constant, Plank’s constant – eight or nine significant decimal digits • most lab measures less accurate than that • Previous computations: • input data – from previous computationla step • may be approximate • beyond our control • determining accuricy expected from a computation

approximations we do have some influence • systematic approximations during computation • Trancation or discretization: • some features of a mathematical model may be simplfied or ommited • e.g., replacing derivatives with finite differences or • using only a finite number of terms in an infinite series

Rounding: • in computations • by hand, with a calculator or a computer • representations of real numbers and • aritmetic operations upon them • ultimately limited to finite amount of presicion • generally inexact

accuricy of final results of a computation • reflect combination of any or all – approximations • resulting perturbations may be amplified • nature of the problem being solved and/or • the algorithm being used • error analysis: • study - effects of such approximations on • the accuracy and the stability • numberical algorithms

Example: Approximations • surface area of the Earch A = 4r2, • number of approximations: • Earch as a sphere – idealization of its true shape • value of radius  6370 km – combination • empirical measurment- previous computation • The value of  ifinite process – trancated at some point • numerical values of • the input data and results of aritmetic operations • rounded in a computer

Broad Classification of Computational Methods • Roots of nonlinear equations • Systems of linear algebric equations • Optimization • Curve Fitting • Interpolation • Integration • Ordinary Differential Equations • Partial Differential Equations • Monte Carlo methods

Roots of nonlinear equation(s) • Roots of nonlinear equation(s) • finding value(s) of a variable that satisfy a single or a set of nonlinear equations f(x) = 0 • problems in engineering design context • mass, energy, force balance, Newton’s laws of motion • analytical solutions • quadratic equations ax2+bx+c=0 • cubic equations – more complex • Abel (1802-1829) proved that no formula existrs for fifth-order poynomials

even such a simple function f(x) = e-x – x = 0 • canot be solved analytically • graphical techniques • plot the function and examine the root(s) visually • rough estimate of roots – lack precision • trial and error: • repeat • guessing a value of x, evaluating whether f(x) • until f(x) is sufficiently close to 0 • inefficient and inadequate formany realstic problems

systematic strategies with computers • simple and efficient • bracketing methods: • start with a guesses that brackets or contains the root • and systematically reduce the width of the bracket • bisection and false position • open methods • trial and error but no guess of a bracket • computationally more efficient but nay not work • e.g., Newton-Rapson and extensions • graphical methods provide inside • Roots of polynomials

Systems of linear algebric equations: • Find values of a vectorial variable that satisfy a set of linear equations Ax = b in matrix form • many problems in verious disiplines • msthematical modeling of large systgms of interconnected elements • such as structures, electrical circuits and fluid networks • other areas of numerical methods • curve fitting and differential equations

direct methods: find the solution in fixed or finite number of computatgional steps • Gaussian elimination • iterative methods: produces a sequence of approximate answers • designed to converge ever closer to the true solution under the proper conditions • Direct meethod – exact result if computations were carried out in an exact aritmetic • the effect of numerical round-off may be significant for large linear systems

for iterative methods: • question of convergence • Do the succesive approximate answers approch to the ture solution? • if so, how quicly? • how should the decision be made to terminate the process?

Optimization • Optimization: • determining value(s) of a scaler or vector variable that corresponds to the “best”: or optimal value of a function – maximum or minimum • engineering design, production planning curev fitting • constraint or unconstraint • linear or nonlinear programming • integer, continuous, mixed integer • dynamic programming • stochastic programming • optimal control problems – determining the best function to optimize a functional

Curve Fitting • fitting curves to data points – regression vs interpolation • regression: significant degree of error associated with data • regression: model input – output relation • uses • prediction and understanding (inference) • linear v.s. nonlinear functional forms • output variable – continuous or categoriacal (classification) • machine learning/ data mining • neural networks, support vector machines • parametric v.s. nonparametric

Interpolation • objective is determine intermediate values between relatively error free data • usual case for tabulated information • The strategy: • fit a curve directly through the data points • use the curve to predict the intermediate values

Integration • geometric interpreation: area under a curve • many other applications • single or multiple integration • finding center of mass • cumulative probability distributions • solution of differential equations

Ordinary Differential Equations • many physical laws - rate of change of some variables • e.g., population growth rate, force lows • initial value and boundary value problems • linear constant coefficient – analtical solutions • linear/nonlinear • single or systems of equations – computation of eigenvalues and eigenvectors • deterministic/stochastic

Partial Differential Equations • characterize systems - the behavior of a physical quantity is expressed as its rate of change with respect to two or more independent variables • e.g., • steady state distribution of temperature on a heated plate (two spatial dimensions) • time variable temperature of a heated rod (time and one spatioal dimension) • two different approaches to solve numerically • finite difference methods: • finite-element methods:

Preliminaries • Nested Multiplication • Review of Taylor Series

Nested Multiplication • some remarks • on evaluating a polynomial efficiently • on rounding and chopping real numbers • To evaluate the polynomial p(x) = a0 + a1x + a2x2 +· · ·+an-1xn-1 + anxn • group the terms in a nested multiplication: p(x) = a0 + x(a1 + x(a2 +· · ·+ x(an-1 + x(an)) · · ·))

Pseudocode • The pseudocodethat evaluates p(x) starts with the innermost parentheses and works outward. integer i, n; real p, x; real array (ai)0:n p ← an for i = n − 1 to 0 do p ← ai+ xp end for

assume that numerical values have been assigned to • integer variable n • real variable x • coefficients a0, a1, . . . , an, - stored in a real linear array. • The left-pointing arrow (←): • the value on the right is stored in the location named on the left (i.e., “overwrites” from right to left) • The for-loop index i runs backward: • taking values n − 1, n − 2, . . . , 0 • The final value of p: • the value of the polynomial at x • nested multiplication procedure: • Horner’s algorithm or synthetic division.

there is exactly • one addition and one multiplication • each time the loop is traversed • Horner’s algorithm evaluate a polynomial with only • n additions and n multiplications • minimum number of operations possible • A naive method - many more operations • e.g., p(x) = 5 + 3x − 7x2 + 2x3 • computed as: p(x) = 5 + x(3 + x(−7 + x(2))) • avoided all the exponentiation operations

p(x) written alternative form: • if n  m • By convention, whenever m < n, define:

Horner’s algorithm - deflation of a polynomial. • process of removing a linear factor from a polynomial • If r is a root of the polynomial p • x − r is a factor of p • The remaining roots of p : • n −1 roots of a polynomial q of degree 1 less than the degree of p: p(x) = (x − r )q(x) + p(r ) • where q(x) = b0 + b1x + b2x2 +· · ·+bn-1xn-1

Pseudocode of Horner’s Algorithm integer i, n; real p, r ; real array (ai )0:n, (bi )0:n−1 bn-1 ← an for i = n − 1 to 0 do bi−1 ← ai+ rbi end for

Introduction to Computational Methods: Approximations in Scientific Computing