Llewellyn J. Cornelius, Ph.D. University of Maryland

1. Using Cultural Competency Assessment Tools as a means of evaluating the reduction of health disparities among persons of color. Llewellyn J. Cornelius, Ph.D. University of Maryland Correspondence: Please address all correspondence to Dr. Llewellyn J. Cornelius, Professor- University of Maryland School of Social Work, 525 W. Redwood Street, Baltimore, MD 21201. Telephone 410-706-7610; Fax 410-706-6046; Email: lcornelius@ssw.umaryland.edu

2. Context: • The need for a culturally competent workforce has been reflected in a multitude of publications, including the both the Culturally and Linguistically Appropriate Services and the National Partnership for Action recommendations published by the U.S. Department of Health and Human Services’ Office of Minority Health (2002, 2010).

3. Purpose: • The purpose of this presentation is to provide an overview of how a consumer-based cultural competency inventory was developed, validated and re-validated using Community Based Participatory Research.

4. Cultural Competency Inventory Validity Process Using CBPR Framework Recognize community as a unit of identity. – The unit of identity was the CCAG (Cultural Competency Advisory Group, a 20-person panel of consumers, therapists, and administrators that advised the Mental Hygiene Administration. The project required their buy-in and participation. Build on strengths and resources within the community. – The priorities of the community as reflected by the CCAG leadership was used to guide the development of the instrument. Israel, B.A. & Schulz, A.J., Parker, E.A., Becker, A.B. (1998). Review of community-based research: Assessing partnership approaches to improve public health. Annual Review of Public Health. 19: 173-202 4

5. Engagement Process Using CBPR Framework (continued) Facilitate collaborative partnerships in all phases of the research. – The presenter and co-presenters’ role was that of providing technical assistance in research methods to the CCAG, while theirs is that of participating in the design of the study and the data collection activities. Integrate knowledge and action for the mutual benefit of all partners. – Knowledge provided by the CCAG regarding the historical and contemporary mistreatment of ethnic populations was critical to the success of the project. Israel, B.A. & Schulz, A.J., Parker, E.A., Becker, A.B. (1998). Review of community-based research: Assessing partnership approaches to improve public health. Annual Review of Public Health. 19: 173-202 5

6. Engagement Process Using CBPR Framework (continued) Promote a co-learning and empowering process that attends to social inequities. –The CCAG facilitated a process that allowed for the discussion of content that would lead to the empowerment of community members. Involve a cyclical and iterative process. – The CCAG meets monthly to address the needs the community. Israel, B.A. & Schulz, A.J., Parker, E.A., Becker, A.B. (1998). Review of community-based research: Assessing partnership approaches to improve public health. Annual Review of Public Health. 19: 173-202 6

7. Validity and Reliability testing process • The following activities were conducted during this period; • 1. Literature review to map out the concepts of cultural competency and satisfaction with the quality of care against the questions in the survey. • 2. Redrafting the survey tool based on the literature review. • 3. Pilot testing the instrument with 50 respondents • 4. Revising the tool based on the pilot test • 5. Translating the instrument into Spanish and Vietnamese.

8. Validity and Reliability testing process cont’d • The following activities were conducted during this period; • 6. Pilot testing the instrument with a sample of 240 respondents in public mental health facilities across the state of Maryland • 7. Conducting and computing correlations between responses to survey questions and conducting an exploratory factor analysis to examine the relationship between measures in the instrument.

9. Validity and Reliability testing process (2001-2002) Literature Review • One of the important factors in the design and implementation of surveys is making sure that each question in an instrument can be tied to some concept or measure. • As such, the team proceeded by conducting a literature review to classify the significant themes in the literature on satisfaction with the quality of care and cultural competency. The team, then, examined the questions in the instrument against these themes, eliminating questions that were not supported by the literature.

10. Validity and Reliability testing process –Literature Review • The following themes emerged from the literature: • Attitude – how consumers feel they are perceived • Communication – issues related to language, being talked with and heard • Treatment – use of healing practices, family involvement and spirituality • Personnel – availability of multicultural staff at various levels • Environment – perceptions of feeling welcomed by staff and agency • Outreach – commitment of staff and agency to engage the community

11. The Validity and Reliability testing process –Instrument Revision • After eliminating questions that were not supported by the literature, the team used principles recommended by Aday and Cornelius (2006) and Sudman and Bradburn (1982) to fine tune the questions. In particular we focused on: • Revisingquestions that had multiple meanings • Examiningthe questions for readability • Shorteningquestions wherever possible • Groupingsimilar questions together • Changing the wording for some similar questions from “agree” to “disagree” to detect inconsistencies in responses to the similar questions. • Formattingthe questionnaire to facilitate response

12. Validity and Reliability testing process Initial Pilot Test (n=50) • After fine tuning the questions, we conducted a pilot test with a convenience sample to focus solely on questions consumers had regarding the clarity of the questions presented and the survey instructions. • After we completed the initial pilot test, we noted that while we achieved the goal of face validity (that is, the instrument validity based on the logic of the instrument) using experts, consumers and the literature to design the instrument, we realized that we can start down the road of conducting more rigorous studies of the validity and reliability testing of the study, if we had a larger sample.

13. Validity and Reliability testing process –Second Pilot Test (n=240) • Utilizing a training manual developed by the consultant, nine members of the CCAG were trained to administer the pilot test. The manual addressed issues such as engaging volunteer respondents, communicating purpose and procedures for the pilot test, discussing confidentiality and risk/discomforts, facilitating group testing and managing collected data (Cornelius, 2001). Of the nine survey administrators, three were proficient in Spanish and one in Vietnamese.

14. Validity and Reliability testing process –Second Pilot test (n=240) • Using claims data from MHP, 13 of the 24 jurisdictions in Maryland, geographically dispersed across the state were selected to assure a balanced regional sampling of respondents of color. These specific jurisdictions also provided the opportunity to sample respondents from urban, suburban and rural areas. Between late January and early April 2002, pilot testing occurred at 30 Psychiatric Programs throughout the state of Maryland. • The empirical examination of the validity and reliability of this instrument followed the administration of the 52 item scale to mental health consumers across the state of Maryland in the January 2002.

15. Validity and Reliability testing process –Second Pilot test (n=240) • Separate analyses of the validity and reliability of the questionnaire items revealed this 52-item scale that had good psychometric properties (Chronbachs alpha=.91) (Cornelius, Arthur, Booker and Morgan, 2003). • Other analyses (correlations and exploratory factor analyses) suggest that there is room for further fine tuning of this instrument. Finally it some of the items in the instrument also lend themselves to other types of validity testing- for example testing the validity of some of the questions regarding the clinical setting via self reported (the current questions) and interview observed measures (new measures) of the clinic setting.

16. A second re-test of the instrument • Following this assessment process, the Mental Hygiene Administration completed a separate re-assessment of the tool to determine whether the instrument would provide consistent findings when examined with another sample of consumers. The instrument was found to have similar findings when re-administered. • Plans are currently underway to pilot the assessment tool in an public mental health care setting as a measure of the delivery of mental health services.

17. Results: The Validity and Reliability testing process Construct Validity Findings TABLE 1: Subscale Intercorrelations, Subscale 1 2 3 4 5 6 7 8 1. Awareness of Patients’ Culture subscale .— .68 .76 .38 .42 .54 .14 .69 2. Respectful Behaviors subscale — .46 .52 .54 .68 .08 .16 3. Language Interpreter Issues subscale .— .29 .31 .41 .22 .51 4. Understanding of Indigenous Practices subscale .— .64 .50 .41 .57 5. Consumer Involvement subscale . — .56 .35 .66 6. Acceptance of Cultural Differences subscale .— .35 .61 7. Community Outreach subscale — .30 8. Patient-Provider-Organization Interactions subscale .—

18. Results: The Validity and Reliability testing process Reliability Findings TABLE 2: Cronbach’s Alpha and Eta-Square Subscale Coefficient Alpha Eta-Square 1. Awareness of Patients’ Culture subscale .69 .16 2. Respectful Behaviors subscale .61 .22 3. Language Interpreter Issues subscale .08 .22 4. Understanding of Indigenous Practices subscale .70 .51 5. Consumer Involvement subscale .72 .45 6. Acceptance of Cultural Differences subscale .52 .23 7. Community Outreach subscale .74 .53 8. Patient-Provider-Organization Interactions .79 .40 Overall score .92 .13

19. Conclusion • The leadership of a consumer led group resulted in the development of an inventory that has good psychometric properties based on the results of the validity and reliability testing of this scale. • Given these findings, it is believed that the overall scale, as well as, one of the subscales can be used to examine consumer perceptions of the cultural competency of their mental health providers. As such, it can be used as a part of a pre-assessment and post-assessment of cultural competency.

20. References • Aday, LA. and Cornelius, LJ Designing and Conducting Health Surveys. (3rd Edition) (2006). San Francisco, Jossey-Bass. • Arthur, TE, Reeves, IG, Cornelius, LJ, Booker NC, Morgan, O, Brathwaite, J, Tufano, T, Allen, K., Donato, I. “A Consumer Assessment Tool for Cultural Competency Within Maryland’s Public Mental Health System.” Psychiatric Rehabilitation Journal, 2005. Vol 28 (3): 243-250. • Cornelius, LJ, Booker NC, Arthur, TE, Reeves, IG Morgan, O,. “The validity and reliability testing of a consumer base cultural competency inventory.” Research on Social Work Practice, 2004, Vol. 14(3) 201-209. • Israel, B.A. & Schulz, A.J., Parker, E.A., Becker, A.B. (1998). Review of community-based research: Assessing partnership approaches to improve public health. Annual Review of Public Health. 19: 173-202 • Sudman, S. Bradburn, N. (1982) Asking Questions.