More on Model Building and Selection (Observation and process error; simulation testing and diagnostics)

1. More on Model Building and Selection(Observation and process error; simulation testing and diagnostics) Fish 458, Lecture 15

2. Observation and Process Error • Reminder: • Process uncertainty impacts the dynamics of the population (e.g. recruitment variability, natural mortality variability, birth-death processes). • Observation uncertainty impacts how we observe the population (e.g. CVs for abundance estimates).

3. Observation and Process Error(the Dynamic Schaefer model) • Let us generalize the Schaefer model to allow for both observation and process error: • determines the extent of process error, and • determines the extent of observation error. • Often we assume that one of the two types of error dominate and hence assume the other to be zero.

4. Process error only-I • We assume here that and continue under the assume that v=0. Under this assumption: • If we assume that w is normally distributed, the likelihood function becomes:

5. Process error only-II • Issues to consider: • A process error estimator can only estimate biomass for years for which index information is available. • The choice of where to place the process error term in the dynamics equation is arbitrary (what “process” is really being modeled?) • You need a continuous time-series of data to compute all the residuals. • What do you do if there are two series of abundance estimates!

6. Observation error only - I • We assume here that and continue under the assumption that w=0. Under this assumption: • If we assume that v is normally distributed, the likelihood function becomes:

7. Observation error only - II • Issues to consider: • An observation error estimator can estimate biomass for all years. • The choice of where to place the observation error term is fairly easy. • There is no need for a continuous time-series of data and multiple series of abundance estimates can be handled straightforwardly! • There is a need to estimate an additional parameter (the initial biomass - often we assume that ).

8. Comparing approaches (Cape Hake) Note that we can’t compare these models because the likelihood functions are different So what can we say about these two analyses

9. Comparing approaches (Residuals) The residuals about the fit of the process error estimator seem more correlated. Formally, a runs test could be conducted. Perhaps plot the residuals against the predicted values; look for a lack of normality

10. Comparing approaches(Retrospective analyses) We re-run the analysis leaving the last few CPUE data points out of the analysis – seems to be a pattern here! Analyses along these lines could have saved northern cod!

11. Simulation testing • Fit the model to the data. • Run the model forward with process error. • Add some observation noise to the predicted CPUE • Fit the observation and process error models. • Compare the estimates from the observation and the process error estimators with the true values. • Repeat steps 2-5 many times.

12. Simulation testing (Cape hake) • The simulation testing framework was applied assuming a process error variation of 0.1 and an observation error variation of 0.2. • The results were summarized by the distribution for the difference between the true and estimated current depletion (the ratio of current biomass to K). • For this scenario, the observation error estimator is both more precise and less biased. Unless there is good evidence for high process error variability (there isn’t for Cape hake), we would therefore prefer the observation error estimator.

13. Simulation testing (Cape hake) Bias - average error isn’t zero

14. Simulation Testing - Recap • The reasons for using simulation testing include: • we know the correct answer for each generated data set – this is not the case in the real world; and • there is no restriction on the types of models that can be compared (e.g. they need not use the same data). • The results of simulation testing depend, of course, on the model assumed for the true situation.

15. Readings • Haddon (2001); Chapter 10. • Hilborn and Mangel (1997), Chapter 7. • Polacheck et al. (1993).