1. Assessing higher education learning outcomes globallyProfessor Hamish Coateshamishc@unimelb.edu.au

2. Are valid and reliable comparisons of learning outcomes scientifically and practically feasible? Study at a glance • An international feasibility study – not a ‘pilot’ • Conceived ~2005, announced 06, scoping 06-09, work 10-13 • Testing 23000 ‘bachelor degree’ students, with data from 5000 faculty, 25 national managers, 250 institutional coordinators, 17 policymakers, and hundreds of advisors and stakeholders • Designed, developed, validated and evaluated frameworks, tests, infrastructure and processes, etc. • HEI reports (not for systems), along with international reports and materials, and recommendations for a main study

3. Linguistically diverse… • Arabic Dutch English Finish • Flemish Italian Japanese Korean • Norwegian Russian Slovak Spanish

4. Instrument architecture Generic Skills Economics Engineering Contextual Dimension National • Is it feasible to develop frameworks and instruments to test discipline-specific learning outcomes? • Test fields: Economics, Engineering Institution Faculty Student

5. Framework and item development Tuning AHELO / QAA frameworks Curriculum documents Accreditation systems Discipline research • Item creation • Gather existing materials • Item workshops • Technical review • Framework mapping • Adaptation, translation • Verification • Outcome specification • Document analysis • Consultation • Synthesis, review • Qualitative testing • Quantitative testing • Operationalisation • Instrument validation • Framework creation Authentic, hybrid item types ‘Above content’ reasoning

6. Translation, adaptation and verification Source version Translation 1 Translation 2 Designed to maintain cross-national comparability of assessment materials A holistic, robust and flexible approach, linked with item production and validation Reconciliation Economists or engineers who are speakers of target language Verification: verifier (linguist) and domain specialist Adaptations managed as a continuous process National review Native speakers of target language trained to detect specific pitfalls Final check

7. Psychometric reviews

8. Student feedback for a task

9. Leadership: International project management and supporting national teams • Operationalisation: Preparing tests and context instruments for secure online delivery, and training makers • Sampling: Engaging institutions, sampling faculty and students, and quality assurance Project implementation • Assessment: Supporting national training, managing testing, and managing marking quality • Reporting: Compiling data products, psychometric and statistical analysis, and system, institution and stakeholder reports • Evaluating: Scientific and practical feasibility, recommendations for full-scale study

11. Country communications

12. Defined engagement cycle established to support systems, institutions and students… but not faculty • Broad insights • Designed sustainable business models for such work… but other models • Assessment frameworks and test instruments developed to support multidimensional test/context instrumentation • Established methods for test design, development, translation/adaptation and validation • Defined operational workflow and quality control procedures required to support global testing • Forming awareness of how such work is positioned globally

13. Institutional positioning Change horizons • System monitoring • Faculty improvement • AHELO • Stakeholder engagement • Policy research • Learner feedback

14. Assessing higher education learning outcomes globallyProfessor Hamish Coateshamishc@unimelb.edu.au