Stages of Concern During Curriculum Change: Formative Evaluation

1. Stages of Concern During Curriculum Change: Formative Evaluation India Broyles EdD and Mildred Savidge PhD University of New England, College of Osteopathic Medicine 11 Hills Beach Rd, Biddeford, ME 04005

2. What is the Difference in Formative and Summative Evaluation? "When the cook tastes the soup, that’s formative. When the guests taste the soup, that’s summative." http://www.designedtoat.com/food2.shtml

3. Learning Outcomes Workshop Objectives The workshop will allow participants to • Value formative evaluation during change • Complete a sample of the SoC questionnaire • Adjust the questionnaire to meet their own innovation or curriculum change • Score the questionnaire and to make preliminary analyses of different types of profiles • Predict ways in which SoC data can facilitate the change process

4. Agenda • 10 minutes – Formative Evaluation of the change process • 20 minutes – Complete the SoC questionnaire with ‘competency-based education’ as the innovation. Score and report. • 15 minutes – Understanding the stages of concern and the SoCQ • 15 minutes – What innovations are you implementing? Rewrite the questions. Exchange with a partner to review/edit the items. Questions? • 20 minutes – Analysis of the data • 10 minutes – What does the college do with the formative evaluation data?

5. Formative Evaluation Formative Evaluation Formative evaluation can serve as a guide for a college in the change process and improves the chances of a successful curriculum innovation that will be both well accepted and long lasting.

6. Formative Evaluation • Formative evaluation is a method of judging the worth of a program while the program activities are forming or happening. • Formative evaluation focuses on the process. • (Bhola 1990).

7. Formative Evaluation typically involves: • gathering information during the early stages of your project or program • a focus on finding out whether your efforts are unfolding as planned • uncovering any obstacles, barriers or unexpected opportunities that may have emerged • identifying mid-course adjustments and corrections which can help insure the success of your work (NW Regional Education Lab) How do we do this?

8. The process of developing and implementing an innovation is Change So we look at the literature on educational change….

9. Individual Innovativeness TheoryCarl Rogers (1995) • Innovators • Early Adopters • Early Majority • Late Majority • Laggards

10. Change is an ongoing process, not a short-term event. “Change requires ongoing support and resources and it takes time. Sometimes new users of an innovation get discouraged when they don't see immediate results. It is important to have realistic expectations about the time it will take to see significant progress and to make sure that other stakeholders in the community understand this as well. Failure to address key aspects of the change process can either add years to, or even prevent, successful implementation.” Gene Hall, Shirley Hord, Archie George. Southwest Educational Development Lab

11. Change occurs in individuals first, then in organizations. The best curriculum in the world will not succeed in your institution unless the people implementing it are ready and willing to make it a success. However, individual change is difficult if the organization is not supportive of the change. Gene Hall, Shirley Hord, Archie George. Southwest Educational Development Lab

12. One Change in Medical Schools… Competency-based Education with emphasis on national core competencies. Please complete the questionnaire. Takes about 10 minutes. Score using quick-scoring guide. Create your profile. You will have 20 minutes total. Give the quick scoring guide to Millie to record. Be sure your name is on it.

13. Agenda • 10 minutes – Formative Evaluation of the change process • 20 minutes – Complete the SoC questionnaire with ‘competency-based education’ as the innovation. Score and report. • 15 minutes – Understanding the stages of concern and the SoCQ • 15 minutes – What innovations are you implementing? Rewrite the questions. Exchange with a partner to review/edit the items. Questions • 20 minutes – Analysis of the data • 10 minutes – What does the college do with the formative evaluation data?

14. What are Concerns? • The world around us is complex. It is not humanly possible to focus at one one time on all of the many different issues. Certain aspects of our world are of higher priority. • The composite representation of the feelings, preoccupation, thought and consideration to a particular issue or task is called a concern.

15. Concerns • Depending on one’s closeness to and involvement with an innovation, one’s concerns will be different in type as well as in intensity. • Many types or levels of concerns can be experienced concurrently. • There is a predictable pattern to the movement of intensity of concern across types.

16. Based on studies at UTEXAS, Austin since 1974 • Stages have been identified with a developmental movement through these stages. • Earlier concerns must be resolved before later concerns emerge. • Timely provision of affective experiences and cognitive resources can facilitate resolution of early concerns and development of higher concerns. • Personalized interventions are important.

17. Stages of Concern Stage 0 -- Awareness. Users have little concern or involvement with the innovation. Stage 1 -- Informational. Users have a general interest in the innovation and would like to know more about it. Stage 2 -- Personal. Users want to learn about the personal ramifications of the innovation. They question how the innovation will affect them. Stage 3 -- Management. Users learn the processes and tasks of the innovation. They focus on information and resources. Stage 4 -- Consequence. Users focus on the innovation's impact on learners. Stage 5 -- Collaboration. Users cooperate with other users in implementing the innovation. Stage 6 -- Refocusing. Users consider the benefits of the innovation and think of additional alternatives that might work even better.

18. Concerns-Based Adoption ModelGene Hall, Shirley Hord, Archie George. Southwest Educational Development Lab • Stages of Concern • Questionnaire • Interviews • Levels of Use • Interviews • Innovation Configuration

19. SoCQ Instrument • Reliability • Coefficients of internal reliability • Test/Retest correlations • Validity • Statistical • Content • Face

20. Coefficients of Internal ReliabilityN = 830

21. Multiple studies: Coefficients of Internal Reliability for each stage

22. Test-Retest Correlations N=132

23. Validity Studies: Statistical • Factor analysis • 7 stages explained 60% of variance • All had Eigen values above 1.0 • Hypothesized scales matched factor scales • Each scale measured an independent construct • Correlations between scale scores shows strong positive relationship between contiguous stages

24. Factor Analysis

25. Correlations between Scale Scores

26. Validity: Comparison Studies • Focus on accuracy in measuring highest and lowest concerns • Interviews and open-ended responses used • Statements matched to scale scores/ profiles • Qualitative analysis showing greatest and least concerns correlated with scale scores • External judges used to establish relationships between interview analysis and scores

27. Validity: Longitudinal Studies • Two 2-year longitudinal studies • Showed concerns changing over time in accordance with concerns theory • Concerns measured by the instrument moved from high Awareness, Informational and Personal concerns to lower concerns on all levels, or higher Consequence, Collaboration and Refocusing concerns

28. Use in clinical education • Lewis and Watson (1997) measured the concerns of 57 nursing faculty about the use of computer technology. Their pre-post study results suggest that the primary concerns of the faculty were informational and that addressing these concerns through workshops increasedinterest in the innovation. • Gwele (1996) measured the concerns of nurse educators (n=93) at four nursing colleges during the implementation of a major legislated curriculum reform. Concluded that when staff is required to adopt a major curriculum change the normal progression through the stages of concern is impeded. They suggested that in these situations it may be important to delay adoption until participants can come to terms with the need to adopt the new curriculum. • Arwer, Harris & Dusold (2004) assessed the concerns of staff during the installation of a telemedicine system and to assure that concerns were addressed during system implementation. Survey findings were used successfully to modify the implementation and training phases of the program to better meet the needs of the staff.

29. Components • Cover Letter / Introductory Page • 35 items • Demographics Create a SoCQ for an innovation in your own institution. (15 minutes)

30. Interpretation of SoC Data Look at your scores/profile on CBE… • Peak Stage Score • 1st and 2nd High Stage Score • Profile Interpretation

31. Peak Stage Scores

32. Group Data • Tally the number of individuals that are high on each stage. This gives a clear picture of the range of peak stage scores within a group (show of hands - how many at each stage). • Aggregate individual data by developing a profile that presents the mean scores of each stage of the individuals.

33. Typical Non-User

34. Institutional Profile of Stages of Concern

35. Profile Interpretation • As individuals move from unawareness and nonuse of an innovation into beginning use and more highly sophisticated use, it is hypothesized that their concerns develop from early to late concerns • Use clinical interpretation techniques • The total score is not generally used in analyses because it does not have a unique meaning.

36. Institutional Profile, 2006

37. Negative One –Two Split

38. Negative One -Two Split with Tailing Up at Stage 6

39. Stages of Concern Stage 0 -- Awareness. Users have little concern or involvement with the innovation. Stage 1 -- Informational. Users have a general interest in the innovation and would like to know more about it. Stage 2 -- Personal. Users want to learn about the personal ramifications of the innovation. They question how the innovation will affect them. Stage 3 -- Management. Users learn the processes and tasks of the innovation. They focus on information and resources. Stage 4 -- Consequence. Users focus on the innovation's impact on learners. Stage 5 -- Collaboration. Users cooperate with other users in implementing the innovation. Stage 6 -- Refocusing. Users consider the benefits of the innovation and think of additional alternatives that might work even better.

40. Intense Management Concerns

41. Consequence Concerns

42. High Collaboration and Consequence Concerns

43. Single High Collaboration Concerns

44. Stages of Concern Stage 0 -- Awareness. Users have little concern or involvement with the innovation. Stage 1 -- Informational. Users have a general interest in the innovation and would like to know more about it. Stage 2 -- Personal. Users want to learn about the personal ramifications of the innovation. They question how the innovation will affect them. Stage 3 -- Management. Users learn the processes and tasks of the innovation. They focus on information and resources. Stage 4 -- Consequence. Users focus on the innovation's impact on learners. Stage 5 -- Collaboration. Users cooperate with other users in implementing the innovation. Stage 6 -- Refocusing. Users consider the benefits of the innovation and think of additional alternatives that might work even better.

45. High Refocusing Concerns

46. High Management Concerns With Ideas

47. Stages of Concern Stage 0 -- Awareness. Users have little concern or involvement with the innovation. Stage 1 -- Informational. Users have a general interest in the innovation and would like to know more about it. Stage 2 -- Personal. Users want to learn about the personal ramifications of the innovation. They question how the innovation will affect them. Stage 3 -- Management. Users learn the processes and tasks of the innovation. They focus on information and resources. Stage 4 -- Consequence. Users focus on the innovation's impact on learners. Stage 5 -- Collaboration. Users cooperate with other users in implementing the innovation. Stage 6 -- Refocusing. Users consider the benefits of the innovation and think of additional alternatives that might work even better.

48. Profile of Impact-Concerned User and Coordinator

49. Unconcerned Innovation User

50. Guidelines • Establish a holistic perspective • Look at High and Low scores • Look at individual item responses • When scores are used in statistical analyses, developers recommend use of raw scores. Conversion to percentiles greatly affects the distribution of the scores.