U.S. Participation in the Assessment of Higher Education Learning Outcomes (AHELO)

1. www.sheeo.org U.S. Participation in the Assessment of Higher Education Learning Outcomes (AHELO) OECD’s Study of the Scientific and Practical Feasibility of Assessing Baccalaureate-Level Student Learning Outcomes Across Nations Association for Institutional Research Orlando, Florida May 29, 2014

2. Why an AHELO? For what purposes? • Official—2006 decision of Council of Education Ministers to examine available options, instruments, practicality, and national/institutional needs • Practical—broad recognition of increasingly global context for higher education students, programs and institutions • Substantive—growing focus on the competitive/economic roles of higher education for nations and students • More consequentially and theoretically, AHELO less a test of “can it be done” than a question of “what purposes can and should it serve” relative to: • Changing student needs? • New international quality expectations and assurances? • Changing global workplace expectations? • Expanding institutional operations and ambitions?

3. National Participants

4. AHELO: Feasibility Study Design Discipline strand in Economics • Subject content and competence • Application to real world problems • Effective application of relevant data and quantitative methods • Communication Discipline strand in Engineering (Civil) • Engineering generic skills • Basic and Engineering sciences • Engineering analysis and design • Engineering application and practice Generic Skills strand • Above content (subject-matter) skills and competencies • Baccalaureate-level preparation for global education and economic environment • Performance task component--assessment of actual application of critical thinking, mathematical reasoning, data interpretation and application, etc. Value-added /“learning gain” strand • Research-based • Indicative of growing interest and underlying evaluative potential • Inadequately resourced

5. AHELO Assessment Instruments Generic Skills Disciplinary Strands—Economics and Engineering Internationally developed instruments with international advisory group One performance task (application in practice) plus discipline content selected response items—45 in Economics and 30 in Engineering Validation through focus groups at participating institutions 90 minutes plus student survey Entirely on-line delivery/data collection Programs or institutions administered tests using international protocols Self-incentives of students and discipline self-interest Scoring by in-country disciplinary faculty using international rubrics Separate on-line faculty survey and institutional contextual data collection • Translation/adaptation of two CLA performance tasks • Additional 25 selected response items in five clusters and multiple domains (from ACER in Australia) • Limited instrument validation—one cognitive lab per country, 5+ students • 120 minutes plus short student survey • Entirely on-line delivery/data collection • Institutions administered/proctored testing using international protocols • Student incentives for participation defined and reported by institutions • Scoring by participating nations using international rubrics /limited QA • Separate on-line faculty survey and institutional contextual data collection

6. What Did the Feasibility Study Accomplish? How successfully? Generally successful instrument development, translation and context adaptation in economics and engineering Challenging first international use of generic skills performance tasks (new to many countries) not easily translated and adapted to non-American contexts, with selected-response (MCQ) questions added later Web-based assessment administration on secure international platforms experienced few technical failures, but encountered linguistic differences and technical constraints (passage length, keyboarding patterns and practices, etc.) Successful completion of pilot test fieldwork in 17 nations, 25 different strand participants, involving nearly 250 institutions and 23,000 students Assessment results scored by participating nations using international guidelines and rubrics, but limited quality controls Extensive international data analysis, but very limited data access and documentation to allow institutional analysis and use

8. U.S. Participants’ Roles • National level • U.S. Dept of Education is member of OECD Education Governing Board • SHEEO—National Project Manager (NPM) and representative on project advisory board--Group of National Experts (GNE) • NCHEMS prepared sample files and will analyze national data • Foundations provided initial funding and remain interested • State level • SHEEO agency provided project leadership, coordination, and oversight in Connecticut, Missouri and Pennsylvania • Institutional level • AHELO Institutional Coordinator and “team” • IR office prepared student/faculty population files • Test Administration—recruitment, scheduling, monitoring • President, provost, faculty, media relations, graduate students

9. US Feasibility Study Results • Institutional participation • Extremely small, voluntary, non-representative sample, but still highly variable participation (hard to generalize domestically or internationally) • Student motivation and participation • Middling at 31% (719/2296), U.S. high of 68% ( highly variable internationally <10% to>95%) • Contextual factors dominate, highly variable across nations and difficult to examine

10. IR Challenges Raised by AHELO • High variability in institutional and national data on students and faculty • Limitations in institutional and system-level analytic capacities and interests • Theory-application gap—Disconnects between psychometric feasibility and the challenges of large scale practical and policy applications • Relevance to teaching and learning • Implications for higher education policy

11. IR Challenges Raised by AHELO • Lack of consistency in institutional and national data on students and faculty • Limitations in institutional and system-level analytic capacities and variable interests • Theory-application gap—Disconnects between psychometric feasibility/requirements and practical limitations/policy applications • Relevance to teaching and learning • Implications for higher education policy

12. Should there be an AHELO be in our future, then—one participant’s perspective • Assessment needs focused purpose clearly understood and communicated • International input and development of assessment design and instruments • Greater involvement of data providers and explicit attention to ultimate users • Stronger international leadership, adequate financing and demonstrable value

13. Thank You Charles S. Lenth State Higher Education Executive Officers (SHEEO) Boulder, CO clenth@sheeo.org AHELO project repository and links: http://www.sheeo.org/projects/assessment-higher-education-learning-outcomes-ahelo