Charlie Lenth clenth@sheeo

1. The Assessment of Higher Education Learning Outcomes (AHELO)OECD’s Study of the Scientific and Practical Feasibility of Assessing Baccalaureate-Level Student Learning Outcomes Across Nations and InstitutionsSHEEO Higher Education Policy ConferenceAugust, 2012 Charlie Lenth clenth@sheeo.org

2. AHELO Context and Development • OECD’s education studies, surveys and analyses • Near universal recognition of changing education needs and objectives, particularly at higher levels • 2006 Council of Education Ministers’ request to plan postsecondary assessment • OECD commissioned papers and convened groups of experts to develop AHELO design

3. Basic Project Objectives and Design • Feasibility study—to determine scientific and practical feasibility of assessing college learning across nations • Assess learning across and within the disciplines or program areas at degree completion • Minimum five diverse nations with non-scientific (convenience) sample of 10 institutions each • Small, random sample of graduating students with institutions as primary unit of analysis • Student, faculty, institutional contextual data to allow meaningful interpretation and analysis

4. U.S. Involvement in AHELO • Considerable U.S. role in research design and development (CAE, Peter Ewell, ETS) • Invitation from OECD to SHEEO, and then to SHEEO members • U.S. participation in Generic Skills Strand since early 2010 and through many months of delays and uncertain funding (and several trips to Paris!)

5. U.S. Financial Support for AHELO Participation • To OECD for assessment development: Lumina Foundation (early and largest foundation donor), SpencerFoundation, Teagle Foundation • For U.S. participation from Hewlett Foundation and U.S. Department of Education: • Payment of national participation fees • Discretionary grant by U.S. Secretary of Education in September 2011 • In-kind contributions by institutions, states, NCHEMS and SHEEO

6. Country Participation in AHELO Generic Skills Economics Engineering Columbia Belgium Australia Egypt Egypt Canada Finland Italy Columbia Korea Mexico Egypt Kuwait Netherlands Japan Mexico Russia Mexico Norway Slovakia Russia Slovakia Slovakia United States Total countries: 17 plus several EU country observers

7. Overview--U.S. Participants’ Roles • National level • US Dept of Education—member of OECD Education Governing Board • SHEEO—National Project Manager (NPM) and representative on project advisory board--Group of National Experts (GNE) • State level • SHEEO agency provides project leadership, coordination, and oversight in Connecticut, Missouri and Pennsylvania • Institutional level • AHELO Institutional Coordinator and “team” • IR office prepares student/faculty files and institutional survey • Test Administration—recruitment, scheduling, set up, test monitoring • Academic leadership—faculty involvement, organizational support • President, media relations, governing board, others • Anticipated analysis and use of returned assessment data

8. U.S. Higher Education Institutions Participating in AHELO Feasibility Study • Connecticut Board of Regents for Higher Education • Southern Connecticut State University (public regional university; 9,000 undergraduates) • Missouri Department of Higher Education • Central Methodist University (private university; 3,500 students) • Missouri State University (public institution; 16,000 undergraduates) • Truman State University (public institution; 6,000 undergraduates) • University of Central Missouri (public regional; 12,000 students) • Webster University (private, not-for-profit; 4,000 undergraduates) • Pennsylvania State System of Higher Education (PASSHE) • Cheyney University of Pennsylvania (public historically black university; 1,300 undergraduates) • Clarion University (public institution; 5,100 undergraduates) • Edinboro University (public institution; 6,600 undergraduates) • Lock Haven University (public institution; 5,000 undergraduates) • Millersville University (public institution; 7,200 undergraduates)

9. Institutional Sampling Planhttp://www.sheeo.org/misc/ahelo/27samplingmanual.pdf • AHELO Sampling Manual provides international guidelines for both student and faculty samples • Basic components: • Institutions provide census of “in scope” students and faculty to NPM (SPSS.sav or Excel .xls/.xlsx preferred) • NPM will draw equal probability samples using prescribed routines and submit for ACER review (generally 200 students and 40 faculty) • Flexibility for stratified or “augmented” sample design if justified and documented by institution (e.g., strata for major program areas, distinguishing student characteristics, special accommodations, etc.) • Review and approval by ACER Sampling Team

10. Generic Skills Assessment Framework/Design • Two rotating performance tasks adapted from Collegiate Learning Assessment (CLA) – 90 minutes • Selected response items from tested Australian item bank – 30 minutes • Brief student experience survey • Faculty/institutional web-based questionnaires • All computer-administered on secure international testing sites

11. Fieldwork, Analysis and Reporting • Test administration by U.S. institutions using secure international testing websites • Lead scorer trained on international scoring rubrics recruited and managed five U.S scorers • Scored assessment results analyzed by ACER and reported to OECD; later to institutions • Preparation, review and release of Feasibility Study findings and recommendations • Final project conference March 2013

12. Context and Consequences • AHELO is part of a larger postsecondary education agenda with implications for students, institutions, nations and the global environment. • Much has already been learned about both scientific and practical feasibility for assessment within an international setting. • OECD and other major organizations will shape future decisions and financial commitments based in large part on this study.