A new exact test to globally assess a population PK and/or PD model C. Laffont & D. Concordet

1. ECOLE NATIONALE VETERINAIRE T O U L O U S E A new exact test to globally assess a population PK and/or PD model C. Laffont & D. Concordet UMR 181 Physiopathologie et Toxicologie Expérimentales INRA, ENVT

2. Classical metrics for global model evaluation • Weighted residuals • WRESNONMEM • CWRESHooker et al. (2007) Pharm Res 24:2187 • PWRESMonolix

3. CWRES WRES Weighted residuals: difficulty in interpretation Karlsson & Savic, 2007, Clin Pharmacol Ther 82:17 Diagnosing Model Diagnostics Simulated data analysed with the correct model !

4. Simulation (PWRES) • FO/FOCE approximation of the model (WRES,CWRES) For linear Gaussian models, Yidistribution is Gaussian so: ! For nonlinear models, WRidistribution is unknown ! Weighted residuals (WR) calculation: • Calculated from the vector of observations Yi in subject i and denoted here WRi Expected distribution if the model is correct ?

5. Expected distribution if the model is correct ? NPDE are assumed to be independent and Gaussian: Independence issue rightly discussed by the authors: when NPDE are dependent, they are jointly not Gaussian Recently proposed metric: NPDE • Simulation-based approach • compares at each time j the distribution of WRijwith their predictive distribution according to the model • Brendel et al. (2006) Pharm Res 23: 2036

6. Reject model with 5% risk Do not reject model = black line (data) outside the ring = black line (data) inside the ring New metric : GUD (Global Uniform Distance) • The purpose of this work was to propose: • an exact test for global model evaluation • an easy diagnostic graph with no subjective interpretation

7. is the Cholesky decomposition of the full variance matrix of Yi WRiare decorrelated within subject i GUD calculation & testing • Step 1 • As for calculation of NPDE, we compute for each subject i the vector of WRi M (=2000) simulations (unbiased)

8. Only true for linear Gaussian models • Case of nonlinear models 1 compt i.v. model ! Decorrelation does not imply independence ! Decorrelation = independence Decorrelation  independence

9. WRi vector (observations) vector ei1 Projectioni1 GUD calculation & testing • Step 2: • To handle data dependency, we use a recentrandomprojection method(See Cuesta-Albertos et al. (2007) for an application) We project WRi vector on R random vectors eir taken from a uniform distribution on the unit sphere r= 1…R Unit sphere

10. WRi vector (observations) We project WRi vector on R random vectors eir taken from a uniform distribution on the unit sphere r= 1…R vector ei2 Projectioni2 GUD calculation & testing • Step 2: • To handle data dependency, we use a recentrandomprojection method(See Cuesta-Albertos et al. (2007) for an application) Unit sphere

11. Subject 1 Subject 2 Subject i Random pdf independent between subjects cdf Mixture of projection distributions GUD calculation & testing Each subject i Projection on R (=100) random directions ...

12. Simulations under H0 95% prediction region GUD calculation & testing • Step 3: • Compare this global cdf obtained for the sample to its distribution under H0 (i.e. correct model) Sample cdf

13. For each replicate, we compute the maximal absolute distance from mean cdf curve (GUD) 95% 0 0 GUD = Global Uniform Distance mean cdf Calculation of 95% prediction region under H0 Simulations under H0 with tested model K =(5000) replicates of the study design Kcdf curves 5% of curves that are the most distant from mean cdf 5%

14. 95% prediction region 95% 0 GUD mean cdf Calculation of 95% prediction region under H0 Simulations under H0 with tested model K =(5000) replicates of the study design Kcdf curves For each replicate, we compute the maximal absolute distance from mean cdf curve (GUD) Uniform region containing 95% of curves = Global Uniform Distance

15. Simulations under H0 P value 95% prediction region Sample Sample cdf 0 GUD test for your sample True model Do not reject model 5% GUD = Global Uniform Distance

16. P value 95% prediction region Sample Sample cdf 0 GUD test for your sample Wrong model Reject model 5% GUD = Global Uniform Distance

17. Sample Sample Do not reject model QQ ring diagnostic plot Reject model

18. = y + y + e Y t 1 2 i i i i i Performances under H0: GUD vs. other metrics • Simulations under H0 to evaluate the level of the tests • 100 subjects i.i.d with 4 obs./ subject • 5000 replications of study design

19. Level of the tests under H0 • Weighted residuals & NPDE calculated in C++ • Kolmogorov-Smirnov (KS) test to test for a N(0,1) Type I error (%) - nominal level = 5%

20. 95% prediction interval for 5000 replicates (Dvoretzky–Kiefer–Wolfowitz) Level of the tests under H0 The p-value should follow a uniform distribution under H0! PP plot Expected value from uniform distribution p value

21. Expected value from uniform distribution p value p value 95% prediction interval for 5000 replicates (Dvoretzky–Kiefer–Wolfowitz) Level of the tests under H0 GUD test PK model PK/PD model Good whatever the level !

22. Expected value from uniform distribution p value p value 95% prediction interval for 5000 replicates (Dvoretzky–Kiefer–Wolfowitz) Level of the tests under H0 NPDE: KS test for N(0,1) PK/PD model PK model  or  of type I error

23. Conclusion • Poor performances of weighed residuals • NPDE show much better performances but do not deal with the issue of data dependency within subjects • Possible increase or decrease of type I error depending on model • New test and graph based on GUD metric • Encouraging results • More work needed to evaluate this test under more complex conditions (different sampling times per subject, real-case data…)