How do measures measure up?

1. How do measures measure up?

2. What is the Centre? Bringing people and knowledge together to promote the best mental health and well-being for every child and youth Knowledge Capacity Partnerships Full continuum of effective and accessible mental health services for children and youth.

3. Brief Introductions • Name and affiliation • Program to be evaluated • One outcome of interest

4. Objectives • To provide an overview of concepts on measurement • To provide guidelines for assessing and selecting measures • To reduce fear on the topic of measures and indicators

5. Outline • What is measurement? • What are the different sources of data and types of measures? • What affects the quality of measures? • How can we select appropriate measures for our evaluation?

6. A. What is measurement?

7. Evaluation Heaven! Getting Ready for Evaluation Develop Data Collection Procedures Review and Select Measures Identify Priority Outcomes Develop Logic Model

8. What is measurement? Evaluation question Measurement refers to the process of “operationalizing” the evaluation question Information needed Sources of information Types of data Evaluation Measure

9. Evaluation question Information needed Sourcesof information Types of data Evaluation Measure Sample Evaluation Question: • What are clients’ perceptions of our services? • Has client satisfaction with our services improved? • What areas of the program do we need to improve on?

10. Evaluation question Information needed Sources of information Types of data Evaluation Measure Information from parents and/or youth on: • Satisfaction with location • Ease in accessing the services • Perceptions of outcomes as a result of the service • Appropriateness of the services • Perceptions on therapeutic alliance or relationship

11. B. What are the different sources of data and types of measures?

12. Levels of measurement • Nominal • Ordinal • Interval • Ratio

13. Sources of information • Questionnaires • Interviews or focus groups • Observation • Administrative data and/or Health records

14. Evaluationquestion Information needed Sources of information Types of data Evaluation Measure Sources of information from parents and/or youth on how they experience our services: • Interview • Focus groups • Administrative data: number and type of complaints • Self-administered survey • Standardized questionnaire

15. Types of Data • Qualitative data: verbal and pictorial • Numeric scores: • Basic units such as frequency, duration, length • Scores from rating scales • Standardized scores (z scores) • Age- or grade-equivalent or adjusted scores • Criteria for evaluating scores: • Standardization: procedural comparability • Psychometric properties

16. Evaluationquestion Information needed Sources of information Types of data Evaluation Measure Type of data on clients’ satisfaction or perception of care • Scores on a standardized measure on client perception of care: • Average of ratings on all items • Changes in average ratings every 6 months

17. C. What affects the quality of measures?

18. Cultural and Historical Context • Use of tests and measures as a “cultural tool” • Cultural appropriateness of the measures • Revisions or updates to measures to accommodate changes in scores in the population

19. Psychometric Properties: Validity Does the measure REALLY measure what it is supposed to measure? Face validity Predictive Concurrent Construct, Criterion and Factorial Content Validity Discriminant or Divergent

20. Psychometric Properties: Reliability = Dependable = Trustworthy = Same old, Same old

21. Psychometric Properties: Feasibility • Cost and availability • Time for administering, scoring and analyzing • Staff involvement • Information management

22. Psychometric Properties: Relevance • Clinical utility • Culturally appropriate • Administrative uses for decision-making • Usefulness for improving program • Usefulness for information on public reports

23. Other measurement considerations when involving children • Developmental level • Educational level • Health status • Family

24. D. How can we select appropriate measures for our evaluation?

25. Criteria for selecting measures • Evidence-based • Feasible • Relevant and meaningful

26. Evidence-based • Sound psychometric properties: valid and reliable • Used in similar settings • Recommended by experts, if no previous literature

27. Sample Summary Tables

28. Sample Summary Table

30. Evaluationquestion Information needed Sourcesof information Types of data Evaluation Measure Youth Services Surveys (YSS, YSSF)

31. Youth Services Surveys (YSS, YSSF) http://www.mhsip.org

32. Evaluationquestion Information needed Sourcesof information Types of data Evaluation Measure Measures of Therapeutic Alliance Blatt & Zuroff 2005: Importance of including therapeutic alliance as a factor moderating treatment outcomes science direct link Elvins & Green 2008: Review of concept and measures on therapeutic alliance science direct link2 Green 2006: Measures on therapeutic alliance for child and youth mental health interscience link Blatt & Zuroff 2005: Importance of including therapeutic alliance as a factor moderating treatment outcomes science direct link Elvins & Green 2008: Review of concept and measures on therapeutic alliance science direct link2 Green 2006: Measures on therapeutic alliance for child and youth mental health interscience link

33. Evaluationquestion Information needed Sourcesof information Types of data Evaluation Measure Measures on Cultural Competency Cultural Competency Self-Evaluation Questionnaire friends link

34. Some common issues • To translate or not • Picking and choosing items from various measures • Making sense of data: indicators, benchmarks and standards

35. Conduct literature review Identify existing measures Examine evidence, feasibility & relevance Use measure Collect & Analyze Data (as Developed) Revise measure Gather new evidence on validity and reliability Identify Gap: NO existing measures Develop new measure Decision process for identifying, selecting and using measures Identifying Selecting Using Specify topic of inquiry

36. Steps in Translating Measures Translate to second language Back translate to first language Pilot-test with experts & end-users Gather new evidence on validity and reliability Compare results with original version

37. Measures Matrix • Evaluation question • Outcome or process variable • Indicator or measure • Where data comes from • How data is collected data & how frequently • Who collects data • When data is collected • How data will be analyzed

38. Sample Measures Matrix

39. Questions?Key insights?

40. Summary • Measurement involves specifying or “operationalizing” the evaluation question. • Process or outcome variables can be assessed using numerical or qualitative data. • Sources of information include questionnaires, interviews, focused groups and administrative data.

41. Summary, Cont. • Summary tables assist in organizing information about various measures, and help in making informed decisions on the most appropriate measure for the evaluation. • A matrix of the indicators and measures to be used in the evaluation summarizes how, where and when data is to be collected.

42. Next Steps • Review logic model and identify evaluation questions • Specify process and outcome variables • Summarize literature review • Review and select measures • Create indicators and measures matrix • Develop protocol for collecting data • Develop plan for analyzing data

43. Future webinars? • February 2009: Using Excel and SPSS for basic statistics • March 2009: Writing the Final Report

44. For more information Evangeline Danseco, PhDHead, Evaluation and Research edanseco@cheo.on.ca613.737.7600, ext. 3319 www.onthepoint.ca