Sensitivity Analysis for Complex Models

1. Sensitivity Analysis for Complex Models Jeremy Oakley & Anthony O’Hagan University of Sheffield, UK.

2. Introduction • An economic model is to be used to predict the cost-effectiveness of a particular treatment(s). • Values of some or all of the input parameters required for the model are uncertain. • This implies the output of the model, the cost-effectiveness of the treatment is also uncertain. • We wish to identify which input parameters are the most influential in driving this output uncertainty. • Should we learn more about these parameters before making a decision?

3. Introduction • Various measures of importance for input variables have been proposed, such as the partial expected value of perfect information (partial EVPI) (Klaxton et al, 2001). • Computing the values of these measures is conventionally done using Monte Carlo techniques. These invariably require a very large numbers of runs of the economic model. • For computationally expensive models, this can be completely impractical. • We present an efficient alternative to Monte Carlo.

4. Sensitivity analysis: partial EVPI • We work with net benefit: utility of treatment is K x efficacy – cost With K the monetary value of a unit increase in efficacy. • The economic model states the utilities of various treatment as a function of a set of input variables X. u= f (X ) • There is uncertainty regarding the true values of the input variables X. • Decision maker chooses treatment with highest expected utility, U *.

5. Value of learning an input variable • Denote Y to be the input variable in the model under question. • Expected value to the decision maker of learning Y before choosing a treatment is U Y. • Value of learning Y, the partial EVPI of Y is U Y- U *. • Partial EVPI is zero if the same decision is made for all values of Y.

6. Improving on Monte Carlo

7. Use each run of the model to learn about the entire model.

8. Efficient computation using an emulator • We can compute partial EVPIs more efficiently through the use of an emulator. • An emulator is a statistical model of the original economic model which can then be used as a fast approximation to the model itself. • An approach used by Sacks et al (1989) for dealing with computationally expensive computer models. • Any regression technique can be used. We employ a nonparametric regression technique based on Gaussian processes (O’Hagan, 1978).

9. Gaussian processes • The gaussian process model for the function f (X ) is non-parametric; the only assumption made about f (X ) is that it is a continuous function. • The interpolation is exact; given sufficient runs of the economic model, the gaussian process emulator `becomes’ the economic model. • It is also possible to allow for uncertainty in the emulator; we can say how our estimate might change if we were to obtain more runs from the economic model. • Additional features can be exploited to speed up computation.

10. Example: GERD model • The GERD model, presented in Briggs et al (2002) predicts the cost-effectiveness of a range of treatment strategies for gastroesophageal reflux disease. • Various uncertain inputs in the model related to treatment efficacies, resource uses by patients. • Model outputs mean number of weeks free of GERD symptoms, and mean cost of treatment for a particular strategy. • Distributions for uncertain inputs detailed in Briggs et al (2002).

11. GERD example • We consider a choice between three treatment strategies: • Acute treatment with proton pump inhibitors (PPIs) for 8 weeks, then continuous maintenance treatment with PPIs at the same dose. • Acute treatment with proton pump inhibitors (PPIs) for 8 weeks, then continuous maintenance treatment with hydrogen receptor antagonists (H2RAs). • Acute treatment with proton pump inhibitors (PPIs) for 8 weeks, then continuous maintenance treatment with PPIs at the a lower dose.

12. GERD example • There are 23 uncertain input variables. • We estimate the partial EVPI for each input variable, based on 600 runs of the GERD model. • We assume a value of $250 for each week free of GERD symptoms. (It is straightforward to repeat our analysis for alternative values). • The GERD model is computationally cheap, and so we can compare our estimates with Monte Carlo estimates based on large samples.

13. GERD Results

14. Conclusions. • The use of the Gaussian process emulator allows partial EVPIs to be computed considerably more efficiently. • Sensitivity analysis feasible for computationally expensive models. • Can also be extended to value of sample information calculations.

15. Acknowledgement We would like to thank Andrew Briggs, Ron Goeree, Gord Blackhouse and Bernie O’Brien for providing us with the GERD model and input distributions.