Introduction to Stochastic Search and Optimization: Random Number Generation Techniques

1. Slides for Introduction to Stochastic Search and Optimization (ISSO)by J. C. Spall APPENDIX DRANDOM NUMBER GENERATION Organization of chapter in ISSO* General description and linear congruential generators Criteria for “good” random number generator Random variates with general distribution Different types of random number generators *Note: These slides cover some topics not included in ISSO

2. Uniform Random Number Generators • Want a sequence of independent, identically distributed U(0, 1) random variables • However, random number generators (RNGs) produce a deterministic and periodic sequence of numbers • What qualities should the generators have?

3. Criteria for ‘Good’ Random Number Generators • Long period • Good distribution of the points (low discrepancy) • Able to pass some statistical tests • Speed/efficiency • Portability – can be implemented easily using different languages and computers • Repeatability – should be able to generate the same sequence over again

4. Generating Random Numbers • Given a transition function, f, the state at step n is given by • The output function, g, produces the outputs as • The output sequence is • Want the sequence period to be close to 2b, where b corresponds to the number of bits

5. Types of Random Number Generators • Linear – most commonly used • Combined – can increase period and improve statistical properties • Non-linear – structure is less regular than linear generators but more difficult to implement

6. Linear Congruential Generators • U(0,1) numbers via linear congruential generators (LCG) are calculated by • These are the most widely used and studied random number generators • The values a, c, and m should be carefully chosen

7. Linear Congruential Generators • Some values for a and m (assuming c = 0) • a = 23, m = 108+1 (original implementation) • a = 65534, m = 229 (poor because of high order correlations) • a = 515, m = 247 (long period, good distribution, but lower order bits should not be trusted) • a = 16807, m = 231 –1 (this has been discussed as the minimum standard for RNGs)

8. 0.58 0.56 0.54 m = 231 – 1, a = 4, c = 1 0.52 Empirical Mean 0.5 m = 482, a = 13, c = 14 0.48 m = 27, a = 26, c = 5 0.46 0.44 m = 9, a = 4, c = 1 0.42 0 500 1000 1500 2000 Number of Samples

9. 30 points 96 points 1 1 0.8 0.8 Uk–-1 Uk–-1 0.6 0.6 0.4 0.4 0.2 0.2 0 0 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 Uk Uk Lattice Structure (Exercise D.2)

10. Fibonacci Generators • These are generators where the current value is the sum (or difference, or XOR) or the two preceding elements • Lagged Fibonacci generators use two numbers earlier in the sequence

11. Multiple Recursive Generators • Multiple recursive generators (MRGs)are defined by where the ai belong to {0,1,…,m – 1} and • For prime m and properly chosen ai’s, the maximal period is mk-1

12. Combining Generators • Used to increase period length and improve statistical properties • Shuffling: uses the second generator to choose a random order for the numbers produced by the final generator • Bit mixing: combines the numbers in the two sequences using some logical or arithmetic operation (addition and subtraction are preferred)

13. Nonlinear Generators • Nonlinearity can be introduced by using a linear transition function with a nonlinear output function • An example is the explicit inversive generator where

14. Random Number Generators Used in Common Software Packages • Important to understand the types of generators used in statistical software packages and their limitations • MATLAB: • Versions earlier than 5: a linear congruential generator with • Versions 5 & 6: a lagged Fibonacci generator combined with a shift register random integer generator with period • EXCEL: un = fractional part (9821×un –1+ 0.211327); period • SAS (v6): LCG with period

15. Inverse-Transform Method for Generating Non-U(0,1) Random Numbers • Let F(x) be the distribution function of X • Define the inverse function of F by • Generate X by • Example: exponential distribution

16. AcceptReject Method • Let pX(x) be the density function of X • Find a function f(x) that majorizes pX(x) • , q is a density function • Generate X by • Generate U from U(0,1) (*) • Generate Y from q(y), independent of U • If , then set X=Y. Otherwise, go back to (*) • Probability of acceptance (efficiency) = 1/c • Related to Markov chain Monte Carlo (MCMC) method (see Exercise 16.4)

17. 2.5 2.0 pX(x) 1.5 q(x) = U(0,1) 1.0 0.5 0 0.2 0.4 1.0 0.6 0.8 1.2

18. U ~ U(0,1): 0.9501, 0.2311, 0.6068, 0.4860, 0.8913,  Y ~ q(y) U(0,1): 0.7621, 0.4565, 0.0185, 0.8214, 0.4447,  : 0.7249, 0.8131,  X ~ PX(x): 0.7621, 0.4565,  accept reject

19. References for Further Study • L’Ecuyer, P. (1998), “Random Number Generation,”in Handbook of Simulation: Principles, Methodology, Advances, Applications, and Practice (J. Banks, ed.), Wiley, New York, Chapter 4. • Neiderreiter, H. (1992), Random Number Generation and Quasi-Monte Carlo Methods, SIAM, Philadelphia.