Quantifying Uncertainty using Classical Methods (Likelihood Profile, Bootstrapping)

1. Quantifying Uncertainty using Classical Methods(Likelihood Profile, Bootstrapping) Fish 458, Lecture 12

2. Quantifying Uncertainty(an overview) • Uncertainty comes in several forms: • Process uncertainty (e.g. recruitment variability, natural mortality variability, birth-death processes). • Observation uncertainty (e.g. CVs for abundance estimates). • Model uncertainty (is the model we chose correct; how many alternative models fit the data adequately?) • Estimation uncertainty – given a model and some data, how well do the data determine the parameters (and predictions) of the model. • Implementation uncertainty – given a management decision, it be enforced?

3. Quantifying Uncertainty(an overview-II) • The various types of uncertainties can be distinguished by: • Can they be reduced by additional research or are they inherent to the system. • Can we quantify them using classical statistical methods. • Today we address “estimation uncertainty”. We defer the other types of uncertainties to future lectures.

4. Estimation Uncertainty • We are going to quantify uncertainty about the estimates of the model parameters (and its predictions of state variables) under the assumption that the model (and likelihood) are correct. • Typical ways to quantify estimation uncertainty include computing standard errors and confidence intervals.

5. Hint: Generic Solver Macro-I Sub ApplySolver(Minpars, TheSheet, TheFunctionValue) Worksheets(TheSheet).Activate SolverReset ' Small precision and automatic scaling SolverOptions Precision:=0.00001, Scaling:=True ' Specify the cell to minimize (MaxMilVal=2) and the parameters to change SolverOK SetCell:=Range(TheFunctionValue), MaxMinVal:=2, ByChange:=Range(Minpars) ‘ Add a constraint (in this case the cells must be positive) SolverAdd CellRef:=Range(Minpars), Relation:=3, FormulaText:=0 ' Don't ask anything at the end of the call SolverSolve UserFinish:=True End Sub

6. Hint: Generic Solver Macro-II • Notes: • Many EXCEL versions do not have all the Visual Basic libraries needed to call SOLVER from a macro loaded. • Within the Visual Basic editor you will need to click “Tools” - “References” - “Solver.xls” to make this library accessible.

7. Likelihood Profile (one parameter) • Fit the model to find the ML parameter estimates and the corresponding negative log-likelihood. • Select a set of fixed values for the parameter of interest. • Minimize the negative log-likelihood fixing the parameter to each value in turn. • Plot the difference between the negative log-likelihood from step 1 and those from step 3.

8. A First Example-I Note: t0=0 and  is assumed known The problem: We are fitting a growth curve to some age and length data. We want to compute the likelihood profile for 

9. A First Example-II We can compute confidence intervals from likelihood profiles Approximate 95% CI

10. Likelihood profiles and confidence intervals • An 100-x% confidence interval for p parameters is determined by finding the values for the parameter(s) for which: • is the negative log-likelihood corresponding to the maximum likelihood estimates.

11. Likelihood Profile (multiple parameters) • Fit the model to find the ML parameter estimates and the corresponding negative log-likelihood. • Select a set of fixed parameter combinations for the set of parameters of interest. • Minimize the negative log-likelihood fixing the values for the set of parameters of interest to each set of values in turn. • Plot the difference between the negative log-likelihood from step 1 and those from step 3 (this creates a surface).

12. What about State Variables-I • Model outputs often include population size, harvest rate, etc. We are usually more interested in these quantities than about the parameters themselves. • However, the state variables are seldom parameters of the model (and cannot be made to be parameters of the model). This makes computing a likelihood profile for them difficult.

13. What about State Variables-II • For each (target) value of the State Variable: • Add a penalty to the negative log-likelihood that increases as the difference between the target value and the model estimate is larger. • It is often a good idea to change the size of the penalty, w, as we get closer to the target (i.e. apply Solver several times, each time increasing w)

14. A Likelihood Profile for Current Biomass

15. Bootstrapping-I • Likelihood profile has some major disadvantages: • Dealing with derived quantities (e.g. state variables) can be difficult. • Dealing with multiple parameters simultaneously is very computationally intensive. • It is impossible to compute likelihood profiles for all of the state variables simultaneously. • These problems can be overcome through bootstrapping.

16. Bootstrapping-II • Bootstrapping deserves a course of its own but, in (very) simple terms, it involves: • Generate a large number of pseudo data sets, each based on the original data set. • Fit the model to each such data set. • Compute summary statistics of interest (standard deviations, confidence intervals, etc.) from the results for each model fit.

17. Bootstrapping III • The most common form of bootstrapping involves developing the pseudo data set by resampling the residuals (with replacement) and adding these to the model predictions: • is the index for year y in pseudo data set U, • is the model prediction of the index for year y, and • y* is selected at random from 1:n.

18. Bootstrapping – IV(the biomass trajectory for Cape Hake)

19. Readings • Hilborn and Mangel, Chapter 7 • Haddon, Chapter 3