Assessing and Assuring Graduate L earning Outcomes

1. Assessing and Assuring Graduate Learning Outcomes Summary of KEY Project findings Presented by Simon Barrie Sydney 26thseptember 2013

2. Curriculum renewal for Graduate learning outcomes Everybody is…. Developing 'new' statements of graduate attribute 'outcomes' Renewing curriculum ……(to achieve such outcomes?) Demonstrating achievements …..(compliance?)

3. What’s it all about? (According to UNESCO) “Owing to the scope and pace of change, society has become increasingly knowledge-based so that higher learning and research now act as essential components of cultural, socio-economic and environmentally sustainable development of individuals, communities and nations.

Higher education itself is confronted therefore with formidable challenges and must proceed to the most radical change and renewal it has ever been required to undertake, so that our society, which is currently undergoing a profound crisis of values, can transcend mere economic considerations and incorporate deeper dimensions of morality and spirituality."

4. Graduate attributes A transformative opportunity? They are descriptions of the core abilities and values a university community agrees all its graduates should develop as a result of successfully completing their university studies (adapted from Bowden et al 2000). Graduate attributes are an orientating statement of education outcomes used to inform curriculum design and engagement with teaching and learning experiences at a university (Barrie 2009). Graduate attributes are one way of starting a rich conversation, across the university and beyond it, about how the university's intentions,values and mission are embodied in the learning experiences and outcomes of its students.

5. Transformative opportunity or managerialist trap? Escher Monkeymen

6. Collaborative national project supported by ALTC / OLT with strategic priority funding • What are the assessment tasks in a range of disciplines that generate convincing evidence of achievement of graduate learning outcomes? • What are the assurance process trusted by disciplines in relation to those assessment tasks and judgements? Situational analysis, literature review, expert reference group, institutional visits, interviews with 48 academics 28 universities in 7 disciplines Ten key issues papers, endnote library, submissions to government discussion papers, research report – 6582 unique page views…and counting. Full story at: http://www.itl.usyd.edu.au/projects/aaglo/

7. What (and why) are we Assessing? • Approach of assessment for learning • Coherent development and assessment of program level graduate learning outcomes requires coherent institutional and discipline statements of outcomes. What that statement ‘is’ matters. • Foundation skills? Translation attributes? Enabling attributes and dispositions? • Traditional or contemporary conceptions of knowledge? • Content knowledge plus generic skills = traditional • Graduate attributes, capabilities, dispositions and stances = contemporary Some of the ways we are articulating GLOs are taking us backwards to outdated old fashioned expectations of the goals of higher education, not forwards to the contemporary outcomes and purposes we espouse.

8. Assessment strategies 2. What we assess, and how we assess it, shapes curriculum and teaching as much as it does learning • Assessment prioritizes some GLO and ignores others • Communication skills – Privileged • Information literacy - Privileged • Research and inquiry – (less) Privileged • Ethical social professional understandings - Neglected • Personal intellectual autonomy - Neglected Consequences: ?

9. Assessment strategies 3. Features of effective assessment practices identified by respondents • (assessment for learning) • interconnected multi-component • authentic, relevant, roles • standards-based with effective communication of criteria • involve multiple decision makers – including students • (program level coherence) • With effective assurance processes around the quality of these tasks and the judgments made

10. Assessment strategies 4. Variety of assessment quality assurance strategies • Quality of task: Pre-implementation (policies, curriculum approval processes for new and revised tasks) Post- implementation (evaluation by students, examiners, peers, curriculum audit & review) • Quality of judgment: Pre-judgement - (calibration) Post-judgement – (consensus moderation) • Confidence? Combination of effective task and assurance was less common than it could be Evidence of ‘assurance’ engaging staff and leading to ‘enhancement’ was rare • Barriers?

11. Assessment strategies – future 4. Trend towards (systematic) program level approach to assessment • Many drivers – complexity of outcomes, coherence & integration, resources, managing regulatory accountability….. others? • Possible changes / implications: longer, bigger, fewer subjects? Limited task variety - cumulative/progressive? Shift from subject focus to assessment of course outcomes (less assessment), integrative assessment, rebalancing of assessment types from FY to graduation? Curriculum design /sequencing on learning challenge not content? • Barriers to making these changes?

12. BARRIErs to change….resources • Cost-effective pedagogies • Those that generate lots of productive studying with modest teaching input (and can still receive very high student feedback ratings) • More feedback …. ? • More teaching ……? • More assessment ……? • Students are an underutilized resource …… active students increase performance at little or no cost • Total teaching hours donot predict student performance … more teaching usually means less time for learning • Increasing students’ social engagement improves retention …. and performance • Library investment in learning resources per student predicts student study hours

13. Figure 1: Different forms of Program-focused assessment (adapted from the Bradford PAsSproject 2012)

14. Read all about itHeadlines from the aaglo issues papers Paper 1: The AAGLO project and the international standards agenda Insight: We are spending a lot of money reinventing the wheel and repeating the same mistakes - but disciplines are coming together around ‘learning’ Paper 2: Assurance of graduate learning outcomes through external review Insight:We have an opportunity to extend ‘peer review’ (in a useful way) to influence curriculum and teaching at an institutional / whole degree level Paper 3: Challenges of assessing graduate learning outcomes in work-based contexts Insight: The boundaries between work and education persist and work is still underutilised as an assessment ground by universities Paper 4: Standardised testing of graduate learning outcomes in higher education Insight:The illusion that standardised generic skills tests usefully define and demonstrate outcomes causes more harm than good Paper 5: Approaches to the assurance of assessment quality Insight: These need to be embedded in academic practice in ways that meaningfully engage staff and students.

15. Headlines from the aaglo issues papers Paper 6: Assessment policy and effective GLO assessment and assurance Insight: Our well-intentioned policy decisions may have a limiting impact on assessment practice Paper 7: Characteristics of effective assessment tasks Insight: Some things really do work! Paper 8: eAssessmentissues in effective GLO assessment and assurance Insight: E-assessment can offer new opportunities through the design of tasks that require Web 2.0 creative activities Paper 9: Whole-of-programme approaches to assessment planning Insight: Fragmented approaches to assessment provide neither a coherent learning experience for students nor credible evidence of GLOs Summary 10:The student perspective on assurance of outcomes Insight: Wekeep leaving the students out of the conversation

16. Curriculum renewal • It is one thing to have the right curriculum goals (learning outcomes) in place….. • ……and it is better to have credible assessment evidence of the achievement of those goals by graduates….. • …..but the real question remains ‘how does a university curriculum, teaching and learning effectively develop those outcomes’?

17. Thank you! Simon.Barrie@sydney.edu.au More at: http://www.itl.usyd.edu.au/projects/aaglo/