Steady-State Statistical Analysis

1. Steady-State Statistical Analysis By Dr. Jason Merrick

2. What We’ll Do ... • Statistical analysis of steady-state simulations • Warm up and run length • Truncated replications • Batching in a single run • Automatic run-time confidence intervals via batch means Simulation with Arena - Steady-state Output Analysis

3. Statistical Analysis of Steady-State Simulations • Recall: Difference between terminating, steady-state simulations • Which is appropriate depends on model, study • Now, assume steady-state is desired • Be sure this is so, since running and analysis is a lot harder than for terminating simulations • Naturally, simulation run lengths can be long • Opportunity for different internal computation order • Can change numerical results • Underscores need for statistical analysis of output Simulation with Arena - Steady-state Output Analysis

4. Warm Up and Run Length • Most models start empty and idle • Empty: No entities present at time 0 • Idle: All resources idle at time 0 • In a terminating simulation this is OK if realistic • In a steady-state simulation, though, this can bias the output for a while after startup • Bias can go either way • Usually downward (results are biased low) in queueing-type models that eventually get congested • Depending on model, parameters, and run length, the bias can be very severe Simulation with Arena - Steady-state Output Analysis

5. Warm Up and Run Length (cont’d.) • Remedies for initialization bias • Better starting state, more typical of steady state • Throw some entities around the model • Can be inconvenient to do this in the model • How do you know how many to throw and where? (This is what you’re trying to estimate in the first place.) • Make the run so long that bias is overwhelmed • Might work if initial bias is weak or dissipates quickly • Let model warm up, still starting empty and idle • Simulate module: Warm-Up Period (time units!) • “Clears” all statistics at that point for summary report, any cross-replication data saved with Statistics module’s Outputs area (but not Time-Persistent or Tallies) Simulation with Arena - Steady-state Output Analysis

6. Warm Up and Run Length (cont’d.) • Warm-up and run length times? • Most practical idea: preliminary runs, plots • Simply “eyeball” them • Statistics module, Time-Persistent and Tallies areas, then Plot with Output Analyzer • Be careful about variability — make multiple replications, superimpose plots • Also, be careful to note “explosions” • Model 5.1: • Run for 1 day (=1440 minutes), 4 replications • Save within-run Shipped parts flow time values Simulation with Arena - Steady-state Output Analysis

7. Warm Up and Run Length (cont’d.) • No explosions • All seem to be settling into steady state • Run length seems adequate to reach steady state • Hard to judge warm-up ... Simulation with Arena - Steady-state Output Analysis

8. Warm Up and Run Length (cont’d.) • “Crop” plots to time 0 - 5,000 • Plot dialog, “Display Time from … to …” • Conservative warm-up: maybe 2,000 • If measures disagreed, use max warm-up Simulation with Arena - Steady-state Output Analysis

9. Truncated Replications • If you can identify appropriate warm-up and run-length times, just make replications as for terminating simulations • Only difference: Specify Warm-Up Period in Simulate module • Proceed with confidence intervals, comparisons, all statistical analysis as in terminating case Simulation with Arena - Steady-state Output Analysis

10. Collecting Truncated Replications • Model 5.1: • Warm-Up Period = 6 hours (=360 minutes) • Run length 1800 minutes (1440 proper + 360 warm-up) • 10 replications • Collect flowtimes for shipped, salvaged and scrapped parts • Statistics module, Outputs area entries to save summary statistics (averages) across replications Simulation with Arena - Steady-state Output Analysis

11. Truncated Replications (cont’d.) • Output Analyzer, Classical Confidence Intervals • Separate invocations due to different units • Interpretation for steady-state expectations here • Want smaller? • More reps, same length • Longer reps, same number of them Simulation with Arena - Steady-state Output Analysis

12. Batching in a Single Run • If model warms up very slowly, truncated replications can be costly • Have to “pay” warm-up on each replication • Alternative: Just one R E A L L Y long run • Only have to “pay” warm-up once • Problem: Have only one “replication” and you need more than that to form a variance estimate (the basic quantity needed for statistical analysis) • Big no-no: Use the individual points within the run as “data” for variance estimate • Usually correlated (not indep.), variance estimate biased Simulation with Arena - Steady-state Output Analysis

13. Batching in a Single Run (cont’d.) • Break each output record from the run into a few large batches • Tally (discrete-time) outputs: Observation-based • Time-Persistent (continuous-time): Time-based • Take averages over batches as “basic” statistics for estimation: Batch means • Tally outputs: Simple arithmetic averages • Time-Persistent: Continuous-time averages • Treat batch means as IID • Key: batch size for low correlation (details in text) • Still might want to truncate (once, time-based) Simulation with Arena - Steady-state Output Analysis

14. Batching in a Single Run (cont’d.) • Picture for WIP (time-persistent): • For observation-based Tallies, just count points • To batch and analyze (details in text): • Statistics module, Time-Persistent, Tally areas to save within-run records (could be big files) • Output Analyzer, Analyze/Batch/Truncate or • Warning if batches are too small for IID • Get means .flt file; Classical C.I. as before Simulation with Arena - Steady-state Output Analysis

15. Collecting Batch Means • Model 5.1: • Warm-Up Period 6 hours (=360 minutes) • Run length 14760 minutes • 10 * 1440 proper + 360 warm-up • Saves 9 warm-ups of 360 minutes from terminating version • Collect flowtimes for shipped, salvaged and scrapped parts • Statistics module, Outputs area entries to save tallies through simulation run • Output Analyzer, Correlogram for 500 lags • Output Analyzer, Analyze/Batch/Truncate Obsn’s • Truncate Time 360, Batch Observations try 150, 200, 250, … • Save batch means to .flt file Simulation with Arena - Steady-state Output Analysis

16. Automatic Run-Time Confidence Intervals via Batch Means • Arena will automatically attempt to form 95% confidence intervals on steady-state output measures via batch means • “Half Width” column in summary output • Ignore if you’re doing a terminating simulation • Uses internal rules for batch sizes (details in text) • Won’t report anything if your run is not long enough • “(Insuf)” if you don’t have the minimum amount of data Arena requires even to form a c.i. • “(Correl)” if you don’t have enough data to form nearly-uncorrelated batch means, required to be safe Simulation with Arena - Steady-state Output Analysis

17. Recommendations, Other Methods for Steady-State Analysis • What to do? • Frankly, try to avoid steady-state simulations • Look at goal of the study • If you really do want steady-state • First try warm-up and truncated replications • Automatic run-time batch-means c.i.’s • Batch-means c.i.’s “by hand” if necessary • Other methods, goals • Large literature on steady-state analysis Simulation with Arena - Steady-state Output Analysis