Three Colleges Collaborate to Assess “Study Abroad” – Influential Factors, Experiences and Outcomes

1. Three Colleges Collaborate to Assess “Study Abroad” – Influential Factors, Experiences and Outcomes Christine Brooks Cote, Bowdoin College James C. Fergerson, Bates College Mark Freeman, Colby College North East Association for Institutional Research Portsmouth, NH November 2004

2. Off-Campus Study Survey Participants Bates Bowdoin Colby Amherst Smith

3. Project Development • Genuine interest in understanding the impact upon students of a semester or a year spent abroad • A need to evaluate the Colby-Bates-Bowdoin Off-Campus Study Programs (CBB) in light of the goals presented to the Mellon Foundation • Seized the opportunity to use the instrument in a collaborative assessment project supported by NEASC and funded by the Mellon Foundation

4. With Support From: “Motivating Inquiry Regarding Teaching and Learning at Critical Transition Points in Liberal Education”

5. “Motivating Inquiry Regarding Teaching and Learning at Critical Transition Points in Liberal Education”20 Liberal Arts Colleges Participated • First Year Seminar study • Senior/Capstone Experience study • Off-Campus Study survey • “Faculty Fellows” interviews of students about focusing on an area of study Institutions could opt in or out of each study

6. Comparing Study Away Programs

7. Survey Development • Based upon a survey designed at Bowdoin • Survey was reviewed and revised by the participating institutions (IR offices, off-campus study offices, faculty, deans, NEASC) • Flexible administration: Common “core” of questions with some local questions; language was localized • Colby/Bates/Bowdoin also used the survey for a final evaluation its CBB Consortium programs in London, Cape Town, and Quito

8. Survey Administration • Bowdoin: Done in conjunction with senior exit interviews • Bates: Stand-alone Web survey • Colby: Web survey in conjunction with Senior Survey • HTML code shared, but some questions and wording were “localized”; data collected locally • Institutions sent data to Colby, for processing/clean-up; SPSS file returned

9. Bates: “Calendar Wars” & “Working the Web” Paper announcement with 5 e-mail reminders . Survey appears as “to do” link in student’s personal academic affairs account until submitted. L.L. Bean gift certificates raffled as “incentives”.

10. Survey Components • Students’ motivations for off-campus study choices • Preparation and advising for study away experience • Student experiences while away • Perceptions of the impact on academic and personal growth • Senior academic activities that built upon the study abroad experience • Several “free-response” questions

11. Major Survey Components Students’ motivations for off-campus study choices Very strong influence No influence on choices on choices • My desire to learn or become more fluent in the language of this country or region of the world • My desire to study in a country or region of the world that is very different from the one I have lived in most of my life • My desire to have a lighter academic load than I would normally have at [Bates or Bowdoin or Colby]

12. Motivations for Study Away (All Institutions) * Not asked by Bowdoin ** “Academic Advisors or Professors” at Bowdoin

13. Major Survey Components Students’ experiences while studying away Strongly agree Strongly disagree • While away, I was part of an intellectual community that was invigorating and thought-provoking. • Most of my time outside of class was spent with people who called the country or region of the world their home. • While away, I nearly always spoke a language other than English.

14. Major Survey Components Students’ perception of impact of off-campus study experience Strongly agree Strongly disagree • The experiences I had while away caused me to change my understanding of how people around the world view the United States. • My understanding of the culture, country or region of the world I was in changed significantly by my living there for a time. • The experiences I had while away taught me a great deal about gender issues facing people in other parts of the world. • Returning to [Bates or Bowdoin or Colby], I engaged in research or independent study that directly built upon the work I did while away.

15. Key Experiences/Outcomes Questions • Did students gain the skills that are desirable in a liberal arts context? • How did outcomes vary by type of student, field of study, or by institution? • Did study away change personal, academic, and career goals? • Did students link study away experiences to other academic work? • How do they compare with those who didn’t study away?

16. Major Study Away Outcomes Overview

17. Other Study Away Outcomes Overview

18. Study Away Reasons - Cluster Analysis • Reasons for studying abroad items are the best candidates for a priori or “causal” variables. • If Reasons presuppose certain outcomes then a model focusing on them has diagnostic value in practice. • Cluster analysis into “types” to simply interpretation. • A fourcluster solution fit the five-school data best

19. Cluster names are subjective interpretations of item response patterns Cluster characteristics Selected Cluster Solution

20. Does Cluster Membership Predict OCS Experiences? Ay-uh… • Well-Roundedsand Experientialistsstrong on cultural immersion, language learning experiences; weakest of the four clusters on “adjusting to academic structure” • Narrow Academics stand out on “keeping in touch with professors back home” and “course learning”, “part of intellectual community”; weakest on cultural immersion experiences • Club Medsweak pretty much everywhere, especially traditional academics – but more cultural immersion than Narrow Academics

21. Does Cluster Membership Predict OCS Outcomes? Ay-uh… • Well-Roundedsstrongest on most outcomes, virtually across the board – “academic integration” as well as “personal growth” and “cross-cultural understanding” outcomes • Narrow Academics weakest on “personal growth” and “cross-cultural understanding” – almost as strong as Well-Roundeds in “integration with post-OCS academics” • Club Medsbring up the rear virtually across the board, especially on post-OCS academic integration. They do, however, report that that the experience was “rewarding”

22. Comparing Students By College By Cluster

23. Academic Division and Clusters - Bates

24. Location of Study – By Cluster (Bates)

25. Cumulative Average GPAs – By Cluster (Bates)

26. Changes in Goals – Bates

27. Core Changes – By Division

28. Linking OCS Survey Data With Senior Survey Outcomes • Responses were linked with HEDS senior survey data (Colby only) – can leverage the OCS survey results against other institution-wide outcomes • And can compare outcomes with non-study abroad graduates

29. Regression Analysis: Are Other Factors Linked To Outcomes? • So, “reasons for studying abroad” account for a healthy portion of the variance in outcomes. • What of other factors? • Sex • Major • Country of study • Does school “uniquely” account for any variation in outcomes? • If so, may help

30. Regression analysis

31. Survey Data Useful for CBB Program Evaluation • CBB Off-Campus Study Programs were funded by the Mellon Foundation • Two critical goals of the programs were to generally enhance the academic quality of study abroad experiences and improve the integration of study abroad experiences with the educational program at the home college.

32. Survey Data Useful for CBB Program Evaluation Motivations The academic opportunities within the program or country support my major CBB 61% All Others 47% My desire to have a lighter academic load than I would normally have at [Bates or Bowdoin or Colby] CBB 5% All Others 13%

33. Survey Data Useful for CBB Program Evaluation Experiences and Perception of Impact While away, I was part of an intellectual community that was invigorating and thought-provoking. CBB 55% All Others 43% I learned a great deal from the courses I took while away. CBB 81% All Others 61%

34. Personal Growth / Reflection “Study abroad gave me the opportunity to have my own thoughts; it provided space from academics that was filled by reflection. That's very important in college because students are confronted by so much information that is difficult to synthesize. With study abroad, there is time and space to reflect on Bates, on the USA, on yourself.” -- Anthropology major, Ghana It is difficult to put in words. I learned so much - and it is still sinking in after a year. I learned to interview powerful political figures in a different language. I learned to live - and enjoy living in a home without indoor plumbing. I even learned a lot from being mugged, getting parasites, and getting sick - all of it opened me up to a different reality from life at Bates. -- Political Science major, Madagascar

35. Connections to Research “Broadly speaking, it was a lesson on life, what it means to be human, and how to interact and communicate cross-culturally. I worked so hard with subsistence campesino potato farmers to produce our own food or kill our own animals, and that has made me much more conscious of the flow of food globally. At Bates I took a Social Justice Spanish class, I was rewarded an Otis Fellowship to return to Bolivia, and I understood my thesis with greater depth. The mix of Bates and Off Campus help me confront my own assumptions and romanticisms about the life in Bolivia, Vermont, and so on. --Environmental Studies Major, Bolivia

36. Senior Thesis

37. Senior Thesis / Research Links “My senior thesis related directly to my experience in Tasmania. The first week of my stay there was the most influential in my thesis. I returned to Tasmania the following summer to do research for my thesis, and having already lived there, made connections and gotten to know the culture and area was very helpful. I could hit the ground running.” -- Environmental Studies major “My thesis was not about Ecuador, but I extensively worked with Spanish texts, so that was important. Although I did not like Ecuador very much, the experience provided me with confidence to apply for a Phillips Fellowship, which allowed me to do pre-thesis research in Cuba.” -- History major

38. Uses of Collaborative Assessment – Advantages • Labor-saving: Divides up workload and tasks • Survey design involved faculty and other administrators • Improved analysis by sharing technical expertise and discussing results • Promoted other data-sharing – Your First College Year results shared • Importance of comparative data – why are we different, and what does that mean?

39. Uses of Collaborative Assessment -- Concerns • Different campus cultures and analytical approaches • Timing • Campus calendars / office work schedules • Assembling the right team(s) • Getting “Buy-in” to exchange data • FERPA concerns (exchanging non-survey outcomes data) • Integrating data from different collection methods

40. Policy Discussions and Faculty Reactions • “This is really interesting!” • At Bowdoin, the faculty Off-Campus Study Committee has reviewed the entire report, including qualitative data from one-on-one and group interviews. • Data from the report will be used in our final CBB report to the Mellon foundation. • Important assessment discussions would focus on reasons behind college differences...yet to happen.

41. Questions? • Contacts: • Christine Brooks Cote, ccote@bowdoin.edu • Jim Fergerson, jfergers@bates.edu • Mark Freeman, mfreeman@colby.edu