Methodological development in biostatistics

1. Methodological development in biostatistics BST 764 – Fall 2014 – Dr. CHARNIGO – Unit One

2. Contents:Selecting research topics… 3Reviewing the literature… 4Theoretical considerations1… 5 – 17Computing considerations2… 18 – 24Practical applications… 25-27Peer review process… 28-30 GENERAL References: 1 Ferguson, T. (1996). A COURSE IN LARGE SAMPLE THEORY. Chapman & HALL: LONDON. 2 TREFETHEN, L. AND BAU, D. (1997). NUMERICAL LINEAR ALGEBRA. SIAM: PHILADELPHIA.

3. Selecting research topics There are several reasons a person may undertake research: • Compelling scientific problem • Inadequacies of existing methods • Developing an innovative idea or new knowledge • Availability of funding • Advisor/colleague expertise • Personal interest/need for credentials What are your reasons for undertaking research ?

4. Reviewing the literature There are several resources a researcher in biostatistics may consult: • PubMed • Google • MathSciNet • JSTOR • Manuscript bibliographies • Conference talks • Others’ recommendations Why is reviewing the literature important ?

5. Theoretical considerations:modes of convergence • Almost surely We say that Xn converges almost surely to X if P[ limn→∞ |Xn – X| <k ] = 1 for any k > 0. How can we visualize this ? When, for example, does convergence almost surely occur ?

6. Theoretical considerations:modes of convergence • In probability We say that Xn converges in probability to X if limn→∞P[ |Xn – X| < k ] = 1 for any k > 0. How is this related to convergence almost surely ? When, for example, does convergence in probability occur ?

7. Theoretical considerations:modes of convergence • In law We say that Xn converges in law to X if limn→∞P[ Xn ≤ x ] = P[X ≤ x ] for all x at which the latter is continuous. How is this related to convergence in probability ? When, for example, does convergence in law occur ?

8. Theoretical considerations:large sample results • Slutsky’s theorem If Xn converges to X and Yn converges to a constant c, then Xn + Yn and XnYn converge in the same mode to X+c and cX. If the mode is in probability or almost surely, or if Xn and Yn are independent, then the conclusion holds with c replaced by Y. Exercise: If Xn converges in probability to µ, then Xn2 – µ2 converges in probability to 0.

9. Theoretical considerations:large sample results • Continuous mapping theorem If Xn converges to X, then g(Xn) converges in the same mode to g(X), provided that P[ X  c(g) ] = 1. Exercise: Let F denote a cumulative distribution function whose inverse F-1 exists and is continuous. If Xn converges in law to U(0,1), then to what does F-1(Xn) converge ?

10. Theoretical considerations:large sample results • Delta method (Cramer’s Theorem) If n1/2 (Xn – µ) converges in law to X, then n1/2 ( h(Xn) – h(µ) ) converges in law to h’(µ) X, provided that h is continuously differentiable in a neighborhood of µ. What happens if h’(µ) = 0 ?

11. Theoretical considerations:magnitudes of random quantities • Op(1) If, for any k > 0 there exists m > 0 such that P[ |Xn| > m ] < k for all n, then we say that Xn = Op(1). Exercise: If Xn converges in law, then Xn = Op(1). The converse need not be true.

12. Theoretical considerations:magnitudes of random quantities • Op(Yn) If Xn/Yn = Op(1) with Yn > 0, then we say that Xn = Op(Yn). Exercise: If Xn = Op(n), then Xn = Op(n2). Exercise: If Xn = Op(n), then Xn-1 need not be Op(n-1). Exercise: A sum of iid random variables with mean zero and finite second moment is Op(n1/2).

13. Theoretical considerations:magnitudes of random quantities • op(1) If Xn converges in probability to 0, then we say that Xn = op(1). How do you suppose we would define op(Yn) for Yn > 0 ? Exercise: Suppose that Zn = Un + Vn, where Un converges in law to U and Vn = op(1). What happens to Zn ?

14. Theoretical considerations:Estimation • Consistency If Xn converges in probability to µ, then we say that Xn is consistent for µ. Exercise: If limn→∞ E[ (Xn – µ)2 ] = 0, then Xn is consistent. Exercise: If n1/2(Xn - µ) = Op(1), then Xn is consistent.

15. Theoretical considerations:Estimation • Unbiasedness If E[ Xn ] = µ, then we say that Xn is unbiased for µ. Which do you think is more important: consistency or unbiasedness ? Exercise: If Xn and Xn2 are unbiased for µ and µ2 respectively, then P[ Xn = µ ] = 1.

16. Theoretical considerations:Testing • Consistency Consider testing H0 against H1. If limn→∞P[ Reject H0 ] = 1 when H1 is true, then we say that the testing procedure is consistent. Exercise: Suppose iid data arise from N(µ, 1) and we test H0: µ = 0 against H1: µ ≠ 0 by rejecting the null hypothesis when n-1/2 ∑i=1n Xi exceeds a fixed threshold in absolute value. Show that this testing procedure is consistent.

17. Theoretical considerations:Testing • Unbiasedness Consider testing H0 against H1. If the infimum of P[ Reject H0 ] when H1 is true is at least as large as the supremum of P[ Reject H0 ] when H0 is true, then we say that the testing procedure is unbiased. Exercise: Suppose iid data arise from N(µ, 1) and we test H0: µ = 0 against H1: µ ≠ 0 by rejecting the null hypothesis when n-1/2 ∑i=1n Xi exceeds a fixed threshold in absolute value. Show that this testing procedure is unbiased.

18. Computing considerations:Speed and memory • Hardware • Software/computing environment • Underlying mathematical problem • Algorithmic What’s “wrong” with this R program that defines a p × p correlation matrix C for variables in an n × p data matrix X ? for (j in 1:p) { for (k in 1:p) { C[j,k] = cor( X[,j], X[,k] ) } } Exercise: For large p and n, the number of computations to obtain C is approximately proportional to np2.

19. Computing considerations:Overflow and underflow Sometimes you will encounter very large or very small numbers, more extreme than your computing environment can accommodate. For example, the value of a likelihood function will be very close to 0 when n is large, so much so that your computing environment may mistake it for 0. This provides another reason to work with a log likelihood, rather than with a likelihood directly. Whereas a computing environment may fail to distinguish a likelihood from 0 when n is large, a log likelihood may still be readily distinguished from -∞. Exercise: Find a number c such that exp(c) is mistaken for 0.

20. Computing considerations:Conditioning Let x denote an input to a mathematical problem, and let f(x) denote the output. Let δx denote a small perturbation to the input, and let δf := f(x+δx) – f(x) denote the corresponding perturbation to the output. Per reference 2, aproblem is said to be well-conditioned if there exists small K > 0 such that { |δf|/|f(x)| } / { |δx|/|x| } < K. A problem that is well-conditioned is one whose output does not change much if the input is changed a little. Exercise: Is subtraction, f(x) = x1 – x2, well-conditioned ?

21. Computing considerations:Accuracy and stability Let f*(x) denote the output from a computer algorithm. Why may the output from a computer algorithm differ from the output of the underlying mathematical problem ? Per reference 2, an algorithm may be called “accurate” if |f*(x) – f(x)| < C |f(x)| for some small C > 0. An algorithm may be called “stable” if |f*(x) – f(x*) | <C|f(x*)| for some x* with |x* - x | <C|x|. Exercise: Subtraction is stable but not accurate. Generally, how are conditioning, stability, and accuracy related ?

22. Computing considerations:User accessibility Several features make an algorithm such as we may implement in R attractive: • No charge for use • Persuasion that the algorithm works “correctly” (providing a convincing example, publishing a paper on the algorithm, making the implementation available at a well-known website) • Documentation inside and/or outside the implementation • Warning/explanatory messages if user’s input is not valid • Speed and stability Why should we wish our algorithm to be attractive to potential users ?

23. Computing considerations:The role of simulation Simulations are often helpful. Roughly speaking, they entail generating fake data sets with known properties, to which you apply your statistical method for a reason such as one of these: • Provide empirical support for a theoretical result. For example, if you have ascertained (or conjectured) that Xn = Op(n-1), you can generate histograms of n Xn for various n and see whether the histograms look similar. Caution: Doing so is not a valid mathematical proof that Xn = Op(n-1), just a plausibility check.

24. Computing considerations:The role of simulation • Identify how large n must be for a theoretical result to be useful in practice. You are familiar with the rule of thumb that the Central Limit Theorem applies when n > 30. When someone develops a new statistical method, however, the sample size at which the new method may be validly applied may be unclear without a simulation study. • Estimate the relationship between power and sample size for a hypothesis testing procedure. Sometimes a simple formula can be written out for approximate power in relation to sample size. However, this may not always be the case. Even if it is, such a formula may not be valid for small n.

25. Practical applications • Sometimes the development of new biostatistical methodology is largely driven by a specific practical application. • Sometimes the methodology itself is of primary interest, but the researcher feels compelled to illustrate the methodology in a practical application. Why do you suppose this is so ? • In any event, modern papers on biostatistical methodology often contain sections in which a “real” data set is analyzed.

26. Practical applications • In reference 8 the authors fit a semiparametric regression model with random effects to relate an index of Parkinson’s disease symptom progression to a subject’s age and a feature derived from a voice recording taken in the subject’s own home. Why do you suppose random effects were needed ? • The idea (see also papers in the bibliography of reference 8) is that telemonitoring of patients with Parkinson’s disease may reduce unnecessary office visits while also signaling the need for urgent clinical assessments. 8. Charnigo, R., Feng, L., and Srinivasan, C. (2014). Compound estimation of partial derivatives in multiple covariates. Submitted for publication.

27. Practical applications • In reference 19 the authors fit normal mixture models to describe birthweight distributions. Although birthweight distributions are roughly normal, there are noticeable departures from normality in the tails. Why should we pay particular attention to the tails ? • In reference 16 the authors fit a beta mixture model to describe a collection of p-values from a microarray experiment. In particular, they estimate the number of such p-values that originate from a uniform distribution. What does a uniform distribution have to do with p-values ? 19. Charnigo, R., Chesnut, L.W., LoBianco, T., and Kirby R.S. (2010). Thinking outside the curve, Part I: Modeling birthweight distribution. Biomedcentral Pregnancy and Childbirth, 10, Article 37. 16. Dai, H. and Charnigo, R. (2008a). Omnibus testing and gene filtration in microarray data analysis. Journal of Applied Statistics, 35, 31.

28. Peer review process • When a manuscript is submitted to a peer-review journal, there are essentially three possible outcomes. The manuscript is accepted for publication. This is rare, unless the manuscript is editorial. The manuscript is invited for revision. Most, but by no means all, of the time, a manuscript invited for revision will eventually be accepted, usually on its second or third version. The manuscript is rejected, either with or without full peer review. For many print journals (but not necessarily online journals), this is the most common outcome. What are some pros and cons of peer review ? What about single and double blinding ? What are some pros and cons of online journals versus print journals ? What influences editors and reviewers to accept (or reject) a manuscript ?

29. Peer review process • If a manuscript is rejected, the authors usually send it to another journal, perhaps after some modifications based on the reviewers’ comments. • If a manuscript is invited for revision, the authors may modify it based on the reviewers’ comments. If substantial modifications are required, they are typically summarized in a response letter that may be sent to the reviewers along with the revised manuscript. Are there circumstances under which declining to revise a manuscript for a journal makes sense ? Is it necessary to address all of the reviewers’ comments ?

30. Peer review process • Requests for extramural funding (i.e., grants) are typically peer reviewed as well. Criteria and procedures differ by funding agency and, to some extent, by program within funding agency. To my knowledge, the following is accurate: The National Science Foundation (NSF) emphasizes Intellectual Merit and Broader Impacts. Possible ratings from individual reviewers are for the proposal as a whole and include “Excellent”, “Very Good”, “Good”, “Fair”, and “Poor”. A committee then makes a recommendation regarding funding. An investigator can re-package a failed application for submission in a future funding cycle. The National Institutes of Health (NIH) emphasize Innovation, Significance, Approach, Environment, and Investigators. Possible ratings from individual reviewers on these categories range from “1” (best) to “9” (worst). A committee then assigns a score between “10” and “90”, and a second committee makes a recommendation regarding funding. One revision of an application is permitted, although an investigator can re-package a failed application for submission in a future funding cycle.