A New Look at Evidence Based Approaches in ADHD Assessment

1. A New Look at Evidence Based Approaches in ADHD Assessment EVIDENCE EVIDENCE EVIDENCE Thomas K. Pedigo Ed.D., NCSP Vann B. Scott, Jr., Ph.D. Ron P. Dumont Ed.D., NCSP EVIDENCE

2. DISCLOSURES • THOMAS K. PEDIGO, ED.D., NCSP • DIRECTOR SAVANNAH CHILD STUDY CENTER, SAVANNAH GEORGIA. • PLEASE NOTE THAT DR. PEDIGO IS THE CO-AUTHOR OF THE PEDIATRIC ATTENTION DISORDERS DIAGNOSTIC SCREENING SYSTEM AND CO-OWNER OF TARGETED TESTING INC. WHICH ARE REFERENCED IN THIS PRESENTATION. • VANN B. SCOTT JR., PH.D. • ASSOCIATE PROFESSOR OF PSYCHOLOGY ARMSTRONG ATLANTIC STATE UNIVERSITY, SAVANNAH GEORGIA. • PLEASE NOTE THAT DR. SCOTT HAS SERVED AS A RESEARCH CONSULTANT TO TARGETED TESTING INC. DURING THE DEVELOPMENT OF THE PEDIATRIC ATTENTION DISORDERS DIAGNOSTIC SCREENING SYSTEM. • RON P. DUMONT, ED.D., NCSP • ASSOCIATE PROFESSOR OF PSYCHOLOGY & DIRECTOR OF THE DOCTORAL TRAINING PROGRAM IN SCHOOL PSYCHOLOGY AT FAIRLEIGH DICKENSON UNIVERSITY, TEANECK NEW JERSEY. • PLEASE NOTE THAT DR. DUMONT DOES NOT HAVE PERSONAL OR FINANCIAL INTEREST IN THE PEDIATRIC ATTENTION DISORDERS DIAGNOSTIC SYSTEM OR WITH TARGETED TESTING INC. WHICH ARE REFERENCED IN THIS PRESENTATION.

3. Dr. Ron P. Dumont

5. Suppose a test was used as a screening measure on a population of 1000 children • in which 4% (40) of the children have ADHD, • and that test gives an abnormal score for 90% of the children with ADHD (i.e., sensitivity) • and gives a normal score for 90% of children without ADHD (i.e., specificity),

6. Sensitivity: proportion of children with disorder who received abnormal test scores

7. Specificity: proportion of children withoutdisorder and who received normal test scores

8. Positive Predictive Power (PPP): probability that one who receives an abnormal test score is correctly classified

9. Negative Predictive Power (NPP): probability that one who does not have an abnormal test score is not classified

11. CONSIDERATION OF BASE RATES • A GIVEN TESTS PREDICTIVE POWER MUST BE CONSIDERED IN LIGHT OF THE GIVEN BASE RATE IN THE POPULATION FROM WHICH IT IS USED. • A TEST WITH .90 SENSITIVITY & SPECIFICITY BASERATE BASERATE BASERATE 04% 50% 100% MAXIMUM POWER MAXIMUM POWER MAXIMUM POWER .20 .90 .40

12. POSITIVE & NEGATIVE PREDICTIVE POWER COMBINED ADHD AND NON-CLINICALS N=200 SINCE WE COMBINED THE GROUPS BEFORE TESTING THE KNOWN CLASSIFICATION OR BASE RATE IS NOW 50% HALF ADHD AND HALF NONADHD. PPP= RATIO OF ADHD SUBJECTS CONSIDERING ALL WHO TESTED POSITIVE. HIGH PPPMEANS THAT A POSITIVE RESULT RULES IN THE CONDITION NPP= RATIO OF NON/CLINICAL SUBJECTS CONSIDERING ALL WHO TESTED NEGATIVE. HIGH NPPMEANS THAT A NEGATIVE RESULT RULES OUT THE CONDITION *ALONG WITH ACCEPTABLE RELIABILITY & VALIDITY, SOLID SENSITIVITY & SPECIFICITY & PPP & NPP HELP TO ESTABLISH THE EVIDENCE NEEDED TO DETERMINE A TEST’S SUITABILITY FOR CLINICAL USE.

13. ADHD POPULATION BASE RATE • HOWEVER, THE BASE RATE FOR ADHD IS WELL BELOW THAT OF THE GROUPS IN OUR EXAMPLES • ESTIMATES VARY WIDELY BUT ARE OFTEN REPORTED BETWEEN 3 AND 7% • FOR OUR WORK AND DEMONSTRATION WE HAVE SELECTED THE CONSERVATIVE ESTIMATE OF 4% • LET’S SEE HOW MEASURES WITH GOOD SENSITIVITY & SPECIFICITY PERFORM WITH THE CONSERVATIVE BASE RATE

14. EBA CALCULATOR

15. GIVEN THE ADHD BASE RATE OF 4% & VARIED PRESENTATIONS RELIABLE AND VALID ASSESSMENT REQUIRES:1.MULTIPLE INPUTS2. EACH WITH ADEQUATE PSYCHOMETRICS3.DEMONSTRATED DIAGNOSTIC UTILITYEX: ACCEPTABLE PPP&NPP4.MUST HAVE CLINICAL AND CONTROL GROUPS

16. INCREMENTAL VALIDITY • TENETS: • MULTIPLE SOURCES OF EVIDENCE USED IN ORDER TO IMPROVE DIAGNOSTIC ACCURACY • THE MULTIPLE INPUTS MUST BE JUSTIFIED IN THAT EACH PROVIDES ADDITIONAL NON-REDUNDANT INFORMATION

17. EBA & LIKELIHOOD RATIOS • 1. ALLOWS INCREMENTAL INPUTS • 2. CAN PROVIDE EVIDENCE FOR OR AGAINST DX • 3. CAN CONSIDER EVIDENCE RELATIVE TO THE KNOWN BASE RATE • ALLOWS YOU TO CONSIDER THE RELATIVE PREDICTIVE POWER OF A TEST’S INDIVIDUAL SCORE POINTS NOT JUST THE OVERALL STATED PERFORMANCE. • 4. USES SCIENCE & CLINICAL EXPERIENCE

18. EVIDENCE BASED ASSESSMENT • HAS BEEN ADVOCATED SINCE MID1990’S & INVOLVES SCRUTINIZING EVIDENCE FOR: • SOUNDNESS • POWER OF INFERENCE • DIAGNOSTIC UTILITY • DEVELOPING AN ATTITUDE OF ENLIGHTENED SKEPTICISM TOWARD DIAGNOSTIC PRACTICES

19. LIKELIHOOD RATIOS • FEW TESTS ARE ACCURATE ENOUGH TO RULE IN OR OUT DIAGNOSIS ALONE. • BEST APPROACH IS TO LOOK AT A GIVEN TEST RESULT AS ALTERING THE PROBABILITY OF AN EXISTENT CONDITION. • REQUIRES THE ESTIMATION OF A PRE-TEST PROBABILITY (BASE RATE)

20. LR CONTINUED • THE PRE-TEST BASE RATE WILL THEN BE ADJUSTED UP OR DOWN BY THE INPUT OF EACH MEASURE/TEST RESULT • ALSO REFERRED TO AS APPLICATION OF BAYESIAN LOGIC. • PRODUCES AN ADJUSTMENT FACTOR < 1 OR > 1 FOR A RANGE OF PROBABILITY FROM 0 TO 99%

21. EBA CALCULATOR

22. LR CONTINUED • CONVERTING TEST SCORE RESULTS INTO LIKELIHOOD RATIOS HELPS DETERMINE HOW USEFUL A DIAGNOSTIC TEST IS • HELPS IN SELECTING A SERIES OR SEQUENCE OF TESTS • CONSIDERS THE RESULTS IN LIGHT OF THE KNOWN BASE RATE • ALLOWS ADDITIVE AND SUBTRACTIVE INPUT TOWARDS THE PREDICTIVE INDEX/OUTCOME

23. STRATEGIES FOR DEVELOPING LIKELIHOOD RATIOS FRAZIER, T.W. & YOUNGSTROM (2006) EVIDENCED-BASED ASSESSMENT OF ATTENTION-DEFICIT HYPERACTIVITY DISORDER: USING MULTIPLE SOURCES OF INFORMATION. JOURNAL AM. ACAD. CHILD ADOLESCENT PSYCHIATRY, 45:5 MAY (2006)

24. STRATEGIES CONTINUED • Web resources for EBA: • (http://www.childrensmercy.org/stats/category/DiagnosticTesting.asp), & • (Centre for Evidence-based Medicine (nd). Likelihood Ratios. Oxford-Centre for Evidence-based Medicine, http://www.cebm.net/likelihood_ratios.asp

25. BASIC STRATEGIES • BASICS: • LITERATURE REVIEW LOOKING FOR SENSITIVITIES & SPECIFICITIES PUBLISHED FOR GIVEN MEASURES • REVIEW CLINICAL MANUALS: • EXAMPLES: CDI, RCMAS, BRIEF, COLOR TRAILS, MANY OTHERS • LOOK FOR TWO GROUPS CLINICAL AND CONTROLS

26. BASIC’S CONTINUED • DETERMINE SENSITIVITY= % TESTING POSITIVE FROM THE CLINICAL GROUP. • FIND THE RELATIVE PERCENTILE OF A GIVEN SCORE POINT(RAW SCORE OR STANDARD SCORE) TO DETERMINE WHAT PERCENTAGE OF THE CLINICAL GROUP FALLS AT THAT GIVEN SCORE POINT. • FIND THE NEXT LOWEST RAW SCORE (SCORE POINT) AND SUBTRACT THE CORRESPONDING PERCENTILE FROM 100% • EX: RS=20 T-SCORE= 50 %= 50TH FIND THE PERCENTILE FOR RAW SORE OF 19 AND SUBTRACT THAT CORRESPONDING PERCENTILE FROM 100%. RS 19=% 45 (100-45= 55) SENSITIVITY OF RS 20 = .55

27. BASICS CONTINUED • SPECIFICITY= THE PERCENTAGE OF SUBJECTS WITH A NEGATIVE TEST RESULT FROM THE NON-CLINICAL GROUP. • WHAT PERCENTILE OF THE NON- CLINICAL GROUP FALLS AT A GIVEN RAW SCORE OR SCORE POINT • FIND THE NEXT LOWEST RAW SCORE AND SUBTRACT THE CORRESPONDING PERCENTILE FROM 100% • EX: RS=20 T-SCORE= 50 %= 50TH FIND THE PERCENTILE FOR RAW SORE OF 19 AND SUBTRACT THAT CORRESPONDING PERCENTILE FROM 100%. RS 19=% 45 (100-45= 55) SENSITIVITY OF RS 20 = .55 • IN THIS EXAMPLE THE SENSITIVITY AND SPECIFICITY ARE EQUAL AT .55 • TO CALCULATE A CORRESPONDING POSITIVE LIKELIHOOD RATIO USE THE FOLLOWING FORMULA • SENSITIVITY/(1-SPECIFICITY) .55/(1-.55) .55/45 = LR 1.22

28. DEVELOPING LR’S FROM RESEARCH DATA • Group a :  # of subjects withADHD, and a positiveTest Score. • Group b :  # of subjects withoutADHD, and a positiveTest Score. • Group c :  # of subjects withADHD, and a negativeTest Score. • Group d :  # of subjects withoutADHD, and a negative Test Score.

29. DEVELOPING LR’S CON’T Sensitivity is the proportion of patients with ADHD who have a positive test. Sensitivity = a / (a + c) Specificity is the proportion of patients without ADHD who have a negative test. Specificity = d / (b + d) Calculate the Ratios: Likelihood ratio (LR+) = sensitivity/(1-specificity) = (a/(a+c))/(b/(b+d)) Likelihood ratio (LR-) = (1-sensitivity)/specificity = (c/(a+c))/(d/(b+d)) . The reference information provided above was adapted from the following Web resources for EBA: (http://www.childrensmercy.org/stats/category/DiagnosticTesting.asp), & (Centre for Evidence-based Medicine (nd). Likelihood Ratios. Oxford-Centre for Evidence-based Medicine, http://www.cebm.net/likelihood_ratios.asp

30. PEDIATRIC ADD SCREENING SYSTEM: (SUMMARY OF THE PADDS INPUTS) • COMPUTER ASSISTED INTERVIEW • ASSESSMENT OF PARENT AND TEACHER DSM-IV RATINGS FOR ADHD • COMPLETION OF THE TARGET TESTS OF EXECUTIVE FUNCTIONING • COMBINED INPUTS TO ESTABLISH A PROBABILITY INDEX • REVIEW OF COMORBIDITY

31. CADI MEDICAL HISTORY/SYSTEMS REVIEW DEVELOPMENTAL HISTORY SOCIAL EMOTIONAL FUNCTIONING DEPRESSION/ANXIETY ATTENTION/HYPERACTIVITY BEHAVIOR/SCHOOL HISTORY

32. Computer Administered/Scored Diagnostic Interview (CADI) EFFECTIVELY ASSESSES FOR COMORBIDITY  ESTABLISHES A PRELIMINARY TREATMENT PLAN  CAN PROVIDE DOCUMENTATION TO SUPPORT REFERRALS AND OTHER TESTING REQUESTS

33. PEDIATRIC ADD SCREENING SYSTEM Target Tests of Executive Functioning (TTEF) ASSESSES EXECUTIVE FUNCTIONS  COMPARES TO ADHD & TYPICAL PEERS  CAN EFFECTIVELY RULE IN & OUT ADHD  EFFECTIVELY CROSS VALIDATES BEHAVIOR RATINGS

34. Barkley’s Model of Behavioral Inhibition Behavioral Inhibition Inhibit Prepotent response Stop an ongoing response Interference control Working Memory Holding events in mind Manipulating or acting on the events Initiation of complex behavior sequences Retrospective function (hindsight) Prospective function (foresight) Anticipatory set Sense of Time Cross-temporal organization of behavior Self-regulation of affect/motivation/arousal Emotional self-control Objectivity / social perspective taking Self regulation of drive and motivation Regulation of arousal in the service of Goal – directed action Internalization of speech Description and reflection Rule-governed behavior (instruction) Problem solving / self-questioning Generation of rules and meta-rules Moral reasoning Reconstitution Analysis and synthesis of behavior Verbal fluency / behavioral fluency Goal directed behavioral creativity Behavioral simulations Syntax of behavior Motor control / fluency / syntax Inhibiting task – irrelevant responses Excluding goal directed responses Execution of novel / complex motor sequences Goal directed persistence Sensitivity to response feedback Task re-engagement following disruption Control of behavior by internally Represented information Used with permission 1/18/2008

38. WHY ADD OBJECTIVE MEASURES • WHY NOT JUST USE RATING SCALES: • SUBJECTIVITY/CAN BE SKEWED • DEMAND CHARACTERISTICS MAY ENHANCE PERSONAL LEANINGS • CAN BE INCONSISTENT EVEN BETWEEN PARENTS AND MULTIPLE TEACHERS • MULTIPLE RATINGS MAY BE CONSIDERED AS REDUNDANT INFORMATION IN SOME CASES • BIAS MAY REFLECT MORE ABOUT THE RELATIONSHIP OF THE CHILD & RATER THAN ABOUT THE ORGANIC FUNCTIONING OF THE CHILD.

39. WHY ADD OBJECTIVE MEASURES • IF OBJECTIVE MEASURES CAN DEMONSTRATE ACCEPTABLE PSYCHOMETRICS (RELIABILITY & VALIDITY), ALONG WITH ACCEPTABLE PPP & NPP,(DIAGNOSTIC UTILITY) THEY COULD SERVE TO ADD INCREMENTAL EVIDENCE (NON-REDUNDANT) FOR OR AGAINST DX. • CAN SERVE TO CROSS VALIDATE THE RESULTS FROM BEHAVIORAL RATINGS • ALLOWS FOR BEHAVIORAL OBSERVATIONS • CAN HELP LOOK AT EFFECTS OF TREATMENT OR INTERVENTION • PARENT AND PHYSICIAN’S MAY MORE READILY ACCEPT RESULTS IF THEY REALIZE THAT MULTIPLE LINES OF EVIDENCED WERE CONSIDERED IN THE ASSESSMENT/DIAGNOSTIC PROCESS. • OBJECTIVE MEASURES FIT WELL INTO A MULTI-METHOD OR EBA APPROACH.

40. Psychometric Properties of the Pediatric ADHD Screener (PADDS) Vann B. Scott, Jr., Ph.D. Armstrong Atlantic State University Savannah, Georgia

41. Clinical sample • 629 (265 females & 364 males) children ages 6 to 12 years (M = 8.66, SD = 1.71) • Data obtained from 10 sites in 7 states

42. Means, standard deviations, standard errors, standard error of measurement, & cut points of three subtests by population Typical and Clinical participants differed significantly on each of the three subtests, all ts >19, p < .001.

43. Reliability estimates

44. Validity Note. Brown Scales and PADDS are inversely scaled therefore, negative correlations are expected.

45. Diagnostic Utility of PADDS relative to other diagnostic tests

46. PADDS SUBTESTS POSITIVE AND NEGATIVE PREDICTIVE POWER CUT SCORES Target Recog= 114 Target Seq= 27 Target Track= 8 DECISION RULE 2/3 IN PREDICTED DIRECTION Sensitivity of .85Specificity of .90 Positive Predictive Power of .90 Negative Predictive Power of .85

47. Incremental Validity Incremental validity – the information added to the diagnostic process through the use of the test beyond what is already known. Incremental validity is demonstrated when the use of a test provides information over and above that demonstrated through other methods for making the same prediction. The PADDS system adds much incremental validity to the diagnosis of ADHD in that it performs as key criterion that provides unique information that improves the accuracy of diagnosis over and above the other sources of information practitioners use to diagnose ADHD.

48. Evidenced Based Assessment – Case Studies • Incremental validity is demonstrated in the PADDS system in the context of the Evidenced Based Approaches highlighted in the following case studies using a variety of diagnostic measures. • Dr. Pedigo will discuss these with you now.

49. CASE STUDY 1 • W/M 8-7 THIRD GRADE • REFERRED FOR ADHD SCREENING • HX OF DISRUPTING CLASS WITH COMMENTS • NOT COMPLETING ASSIGNMENTS • MAKES CARELESS ERRORS • DIFFICULTY COMPLETING HOME WORK • RECENT DROP IN GRADES B’S TO C-D RANGE

50. CASE STUDY 1 MEASURES • PROCEDURES USED: • PARENT INTERVIEW (BACKGROUND) • CHILD INTERVIEW (BK & OBSERVATION) • RIAS) (IQ SCREENING) • WRAML-2 (IMMEDIATE MEMORY) • PADDS (SCREEN COMORBIDITY) • (OBJ ASSESSMENT EF) • BRIEF (PAR/TEA RATINGS OF EF) • BASC-2 (ASSESS SEVERITY OF C0-M) • EDDT (SCREEN EBD)