Engagement: Tensions and Solutions

1. Engagement: Tensions and Solutions Dr Colin Mason Dean, Teaching and Learning

2. FORMATIVE ASSESSMENT RESEARCH LEARNING QUALITY MASS DIVERSE APPLIED INNOVATION ENGAGEMENT ELITE HOMOGENOUS TRADITIONAL LECTURES INDUCTION ACCOUNTABILITY EVALUATION SUMMATIVE ASSESSMENT TEACHING Tensions in Tertiary Education

3. Outline • Some of the tensions in tertiary education (worldwide) • Case Studies: • Scottish Approach to Quality assurance (ELIR and Enhancement themes) • Research and Teaching linkages • Assessment practice • A case for individualised student support? • A model for solutions to tensions in continua • Staff engagement too

4. Teaching Quality – Enhancement-led Institutional Review (ELIR) • Universities monitor quality themselves at subject level • Periodic review by QAA • Different systems in England and in Wales • Scotland better? • Emphasis on Enhancement

5. Scottish Peer Review 5 Yr Visits • Visit Team 5 people including • 3 Academic staff, • 1 student, • review secretary- administrator • Process • Reading documentation • Interviewing students/staff who come to the reviewers • No visits to individual Schools or Units

6. SFC/QAA Approach to Enhancement Themes (2003-4) • Institutional focus groups and surveys • good practice • development needs • Identifying international best practice to meet development needs identified through the surveys • National and International experts invited to share their expertise widely in Scotland • One-two day National conference for students/staff

7. QAA (Scotland) Enhancement Themes • 2003 • Responding to Student Needs • Assessment • 2004 • Employability • Flexible Delivery • 2005 • Integrative Assessment • The First Year • 2006 • The First Year • Research-Teaching Linkages

8. Booklet and 2 Volumes published: 17 June 2005 STOP PRESS Assessmenthttp://www.qaa.ac.uk/scottishenhancement/events/default.htm Review of degree classification: • Assessment seminar: Considering the Honours degree classification system - 11 May 2004 8 Workshops on Good practice • (Jan – June 2004) • Key note (international) presentations • Case Studies (Scottish) • Discussion groups Book

9. Assessment Workshops 1 - Streamlining assessment - How to make assessment more effective and more efficient - 13 January 2004 2 - Using assessment to motivate learning - 5 February 2004 3 - Constructive alignment of learning outcomes to assessment methods - 27 February 2004 4 - Developing a variety of assessment methods, including self and peer assessment - 19 March 2004 5 - Assessing online - 16 April 2004 6 - Issues of validity, reliability and fairness - 7 May 2004 7 - Improving feedback to students (link between formative and summative assessment) - 4 June 2004 8 - Assessment of Personal Transferable Skills - 29 June 2004

10. EVOLUTION QAA (Scotland) Enhancement Themes • 2003 • Responding to Student Needs • Assessment • 2004 • Employability • Flexible Delivery • 2005 • Integrative Assessment • The First Year • 2006 • The First Year • Research-Teaching Linkages ‘Expertise’ delivered to the sector Projects undertaken by the sector

11. How this affected student learning quality at St Andrews • Learning outcomes, curricula and assessment, including programme specifications • The student learning experience • Mentoring for staff • Peer observation of teaching • Funded Innovation • Raising the status of teaching?

12. “Without all these (UG) students how could I do any research” Tension between Research and Teaching “If it wasn’t for all these (UG) students I could do more research”

13. Research-Teaching Linkages • Boyer Commission • Reinventing undergraduate education: A blueprint for America’s Research Universities (1995) • 10 ways to change undergraduate curriculum

14. Boyer’s Ten ways to ….. • Make research-based learning (“mini-projects”) the standard (involve UGs in research process; mentor for every student; internships) • Inquiry-based first year (“curiosity, problem-based learning; including group projects”) • Building in the second year (including integration of direct entry) • Remove barriers to inter-disciplinarity (flexible course ‘swaps’)

15. Boyer’s Ten ways to ….. • Link communication skills and course work (“Presentations and writing – ‘down’ as well as ‘up’”) • Use Information Technology creatively eg electronic classrooms (“exploit digital technology and media”) • Culminate with a capstone experience – (“compulsory project or dissertation”)

16. Boyer’s Ten ways to ….. • Graduate students as apprentice teachers (“Postgraduate tutors”) • Change faculty reward systems – highest standards in teaching and research (“and reward systems to match”; evaluate teaching – look for integration of research with teaching) • Cultivate a Sense of Community (“strong social as well as academic opportunities”; see diversity as an asset)

17. Teacher Focus Research Process Research Content Student Focus A model of Research-Teaching linkage in curriculum design Griffiths, Brew, Healy: 2004-2005

18. Some practical ways forward • Jenkins and Healey (2005) • Develop research policies and strategies to strengthen the link • “Schools (Departments) are expected to identify research policies that support the curriculum and underpin teaching (Gibbs, 2003)” • Mason 2006 (ELIR visit) • Fund for Innovation in Learning, Teaching and Assessment (FILTA); criterion for project bids that explicitly link how author’s research will be integrated with the teaching innovation

19. An interlude (with a purpose)

20. Q. Write down the names of 10 colours 8.5 A. 1 Black 2 White 3 Yellow 4 Blew 5 Green 6 Red 7 Orrange 8 Indigo 9 Broon 10 Violet 9 7 10 5 0 Almost any number (0-10) Score this answer out of 10

21. Assessment

22. CREATIVITY ORIGINALITY EXAMINATIONS VALIDITY SUMMATIVE HOLISTIC GROUP INDIVIDUAL ATOMISTIC FORMATIVE MASTERY RECALL RELIABILITY COURSEWORK TUTOR ASSESSOR PEER ASSESSOR Tensions in Assessment

23. DIAGNOSTIC DIAGNOSTIC FORMATIVE FORMATIVE SUMMATIVE SUMMATIVE Forms (Modes) of Assessment ‘Reality’ ‘Theory’

24. Complex outcomes of learning • ‘Complex’ learning outcomes including: • higher order academic abilities (analysis, synthesis, evaluation) and • ‘soft skills’ (eg teamwork, communication, time management) • are rarely, and often poorly and inconsistently defined. • Such learning is: • Advanced and difficult to ‘measure’ • Slow to develop (100s or 1000s of hours?) • Fuzzy (precision & reliability often only possible at the expense of validity).

25. Myths about assessment • Assessment ≠ measurement • Problems with summative assessment (Knight 2002) • Limits of number • High-precision, high reliability • Threats to validity • Representing all aspects of student achievement • So ……

26. Feedback (Formative) Student disclosure Feedout (Summative) Student deception Shift to Formative Assessment • Knight, PT (2002). “Summative Assessment in Higher Education: practices in disarray”. Studies in Higher Education, 27, No.3, 275-286. Assessment

27. A shift in the balance of modes of assessment? • Low-stakes or formative purposes • Create informed feedback on achievement in order to improve future achievement: • SENLEF Higher Education Academy and Scottish Higher Education Developers • http://www.heacademy.ac.uk/ourwork/learning/assessment/senlef • REAP, Scottish Funding Council 2005-07 (D Nicol, Strathclyde) • http://www.reap.ac.uk/ • Specifically think about ‘Feed-forward’ as well as feed-back

28. How to increase ‘feed-forward’? • Encourage students to submit drafts – provide feedback and then they re-submit • First years are particularly vulnerable • What about having no summative assessments in Semester 1? • For coursework: • Apply progressive weighting to assignments to minimise impact of adjusting to a new environment

29. A model for assignment weightings to improve ‘feed-forward’ Coursework Examination Assignment 2 Assignment 3 Assignment 1 40% 10% 20% 30%

30. Summary • High-stakes, summative assessment has limitations for ‘complex’ learning • Reliability & precision is expensive • Low-stakes, formative assessment is good for learning • Low stakes can reach achievements that elude high-stakes approaches (eg teamwork) • Blended assessment: preferably planned at a programme level.

31. So, what about the first year? • Induction is a problem for anyone new to an institution • Cultural differences may exacerbate the problem • What is happening in the UK (Scotland)?

32. When you are in deep shit, say nothing, and try to look like you know what you are doing

33. Entering the university: the differentiated experience of two Chinese international students …. • Longitudinal study on a single course (module?) and their personal experiences to establish an identity • B. Business Studies, 12 students enrolled, 2004 • 3x semi-structured interviews – Semester 1 (& follow up) • 2 groups emerged: • Negotiated a pathway for survival • No obvious headway in new setting

34. Capturing the journey Before I come I take IELTS and I think, ‘Oh, they speak English and study in that is very good’ ‘But when I come to here many things is different’ .. Different learning and no people to help you in the English .. And you don’t know what it means ‘Mike’

35. New approaches (when expected to learn for themselves -DIY) .. When I come I feel I can taste a little bit sweet of the successful, because I achieve a goal and I never give up, and keep trying, keep trying, keep trying .. Before the lecture I read the books and see the WebCT and then the tutor say anything, I know. Because I read the books, textbooks. And I think it is not difficult. ‘Saul’ ‘Mike’

36. A reality check – the first test .. Actually, I have no point about what is the lecturer want us to catch … I don’t know which part I should focus on, should concentrate. .. The first few weeks of study I have no idea. I haven’t read the book and I just attend the lecture and then listen and no [no comprehension]. At home I have read no books ‘Saul’ ‘Mike’

37. Mike finds a strategy (how to learn) .. After the test I thought, oh, the lecturer just ask us to focus on this .. That is teacher’s way to test us which part is important and which is unimportant.’ .. I borrow my friend’s last year’s test, I saw that and, oh, this question relates to the textbook and my notes and I should read my notes and text carefully with the lecture notes! ‘Saul’ ‘Mike’

38. Both fail the exam but … Mike reflects on why; Saul is bewildered .. ‘You know, I go to every workshop and I go to every lecture and I take marvellous notes, and before the tests I used one week to review.’ ‘I think whole semester is quite bad’ .. ‘I realise when I read the text or books I think [if] I understand, I can remember, so I just general read the books and then actually I can’t remember.’ ‘Saul’ ‘Mike’

39. Author’s interpretations • Both students tried to meet course demands • Both used methods based on previous experience • Both were confused • Both were able to reflect but … • Mike was able to develop new insights and developed a sense of self-efficacy

40. Author’s interpretations • Large first year classes were not conducive to approaches in previous language learning classes • So … • Mike recognised the value of reading before lectures (and he recognised he needed to work harder) • Saul was pre-occupied with discovering what was important or necessary and less with how necessary it was to understand (deeply)

41. Author’s conclusions • The institution and staff did not have good systems in place to support these students. • Even for Mike, smaller groups would be necessary, with early opportunities to engage with teaching staff • Every Cloud …. • Saul repeated the course ….

42. Saul finds the silver lining .. ‘Because this is my second time to study this course so I already know all things and all the key point and all the stuff which the lecturer want to ask to understand it, you know, so our study group, it’s like I’m the leader of that.’ .. ‘Oh, Saul said this’. ‘Oh, that’s very important, better write that down.’ Saul thinks …’ You know, like little teacher, you know. ‘Saul’

43. Entering the university: the differentiated experience of two Chinese international students in a New Zealand University [Massey] Gillian Skyrme Studies in Higher Education, 32, 3, pp357-372, 2007.

44. QAA Enhancement Theme:The First Year - Projects • Curriculum design for the first year • Formative and diagnostic assessment and feedback • Peer support in the first year • Personal development planning • Personalisation of the first year • Introducing scholarship skills • Transition £10,000; Collaborative, multi-institutional

45. Conclusions • Engagement is (and always will be) about curriculum design • What appear to be polar opposites aren’t necessarily – consider ‘horse-shoe’ models? • Staff engagement is as vital as student engagement

46. So, how is Unitec doing?

47. Finally, some ways to ‘pull’ more staff along with you • Fundedprojects: to promote research-linked, teaching innovation and professional development • Prizesfor teams, as well as individuals, to develop innovative approaches • Disseminate good practice; and develop ‘others’ (mentoring and coaching colleagues) as well as self • Accountability • Performance review based on evidence (with follow-up) • Report on professional development opportunities • Academic (e)-portfolios and profiles for promotion (and prizes) • CTLI Support & Guidance: • School/individual focus – Teaching symposia; Mini-symposia ‘Away-days’; Communities of Practice; informal ‘conversations