1 / 61

Dr Sue Taylor*, Dr Mary Ryan**, Dr Jon Pearce#

Utilising E-Technology, Peer Review Feedback and Reflective Practices to Reposition Students as Responsible Partners in Their Own Learning within a Mass Education Context. Dr Sue Taylor*, Dr Mary Ryan**, Dr Jon Pearce#

idola
Télécharger la présentation

Dr Sue Taylor*, Dr Mary Ryan**, Dr Jon Pearce#

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Utilising E-Technology, Peer Review Feedback and Reflective Practices to Reposition Students as Responsible Partners in Their Own Learning within a Mass Education Context Dr Sue Taylor*, Dr Mary Ryan**, Dr Jon Pearce# *School of Accountancy, QUT Business School, QUT, Gardens Point Campus, Brisbane, Australia; **School of Curriculum, Faculty of Education, QUT, Victoria Park Road, Kelvin Grove Campus, Brisbane, Australia; and #Department of Information Systems, University of Melbourne, Victoria, Australia

  2. Presentation: Extracted From Submitted Paper as Detailed Below – Literature References Utilised in This Presentation Can Be Located Within This Paper Paper ID: 784 Session: New challenges for the Higher Education Area Presentation type: VIRTUAL Title: UTILISING E-TECHNOLOGY, PEER REVIEW FEEDBACK AND REFLECTIVE PRACTICES TO REPOSITION STUDENTS AS RESPONSIBLE PARTNERS IN THEIR OWN LEARNING WITHIN A MASS EDUCATION CONTEXT View abstract Authors: J. Pearce; University of Melbourne (AUSTRALIA)              M. Ryan; Queensland University of Technology (AUSTRALIA)             S. Taylor; Queensland University of Technology (AUSTRALIA)

  3. Introduction and Background External and Internal Tensions in Higher Education: • At the international level, the higher education sector is currently being subjected to increased calls for public accountability and the current move by the OECD to rank universities based on the quality of their teaching and learning outcomes. • At the national level, Australian universities face challenges including: financial restrictions, increasing student numbers, time-poor academics and students and the resulting fragmentation of academic programmes across flexible learning options.

  4. Introduction and Background (cont’d) Response of the Australian Higher Education Sector and the Negative Impacts on Student Learning • The Australian higher education response to these competing policy and accreditation demands focuses on precise explicit systems and procedures which are inflexible and conservative. • These bureaucratic practices undermine student achievement by failing to acknowledge that high level and complex learning is best developed when assessment, combined with effective feedback practices, involves students as partners in these processes.

  5. Introduction and Background (cont’d) Response of the Australian Higher Education Sector and Negative Impacts on Student Learning (cont’d) • Universities must acknowledge that both self and peer assessmentenable students to monitor and evaluate the quality and impact of their own work and that of others. • Without these graduate capabilities students face a lifelong dependence upon others.

  6. Is There A Resolution?: The Role of Innovative Assessment Design Innovative assessment design, which includes new paradigms of student engagement and learning and pedagogically based technologies, has the capacity to provide some measure of relief from these internal and external tensions. The key to this positive outcome is to significantly enhance the learning experience for an increasingly culturally and educationally diverse and time-poor population of students as highlighted by Huijser, Kimmins, & Evans.

  7. Is There A Resolution?: The Role of Innovative Assessment Design (cont’d) This social constructivist view highlights the importance of the peer review process in assisting students to participate and collaborate as equal members of a community of scholars with both their peers and academic staff members and to thus become responsible partners in their own learning. • It is also important to note that the current thinking around self and peer-assessment highlights the need to broaden these self and peer-review concepts to include a reflective practices stance (Ryan & Ryan, 2012a). • That is, in order to achieve high levels of ‘active engagement’ by students, rigorous reflective learning processes need to be deeply embedded within the peer review process and carefully and explicitly scaffolded for students (Ryan & Ryan, 2012a). 

  8. Is There A Resolution?: The Role of Innovative Assessment Design (cont’d)

  9. Is There A Resolution?: The Role of Innovative Assessment Design (cont’d)

  10. Is There A Resolution?: The Role of Innovative Assessment Design (cont’d) Engaging students with assessment within a collaborative group process, encourages peers to act as positive role models within a non-intimidating, informal environment. Thus, any discussion of the use of peer review within an e-learning environment is both important and timely.

  11. Study Motivations and the Purpose of Results Dissemination: Case Studies One, Two and Three • As highlighted by Taylor (2011 (a) and (b), the initial Case Study One and Two journeys into the peer review process began with the creation of peer-reviewed assessment tasks within both the Financial Accounting and International Accounting under-graduate subjects in the School of Accountancy, QUT Business School. • This innovative process was designed in recognition of the fact that assessment is the single biggest influence on how students approach their learning.

  12. Study Motivations and the Purpose of Results Dissemination: Case Studies One, Two and Three (cont’d) • As highlighted by Topping (at p. 20): “...the primary objective of peer assessment then is to foster engagement through a peer review process which...helps students help each other plan their learning, identify their strengths and weaknesses, target areas for remedial action, and develop...other personal and professional skills.”

  13. Study Motivations and the Purpose of Results Dissemination: Case Studies One, Two and Three (cont’d) • Of importance to note and as will be detailed later in this presentation, to actually achieve effective student engagement while simultaneously benefitting time poor staff and students, the concepts and practices of self and peer-review must be broadened to include both a collaborative, on-line technology process and a reflective practices stance.

  14. Study Motivations and the Purpose of Results Dissemination: Case Studies One, Two and Three (cont’d) It is anticipated that the benefits of the e-Learning-based, peer review process will beALL the relevant stakeholders. That is: • Students - will engage in collaborative and reflective practices which serve to develop life-long learning skills; • Accounting Educators – will develop new skills as mediators and moderators; • QUT – will benefit from the constructive alignment of assessment tasks – utilising minimal resources; • The Accounting Profession – will significantly benefit from employing post-peer-review students who have increased technical accounting and technology skills and non-technical skills in communication, teamwork, problem-solving and self-management; and • The Broader Community of Accounting Educators – will benefit from the creation of an increased dialogue related to new and, potentially, more relevant modes of student assessment.

  15. What is Student Peer Review and What Does it do for Student Learning? Benefits of Peer Review – 1) Social Constructivism • Social constructivism highlights the critical importance of the social context of learning, emphasising the role of both teacher and learner in the development of complex cognitive understandings and the generation of new knowledge (Vygotsky, 1962; Adams, 2006). • Importantly, consensus between individuals is held to be the ultimate criterion upon which to judge the veracity of knowledge. Peer learning and assessment is one such context that values consensus of quality from members of a community of learners, rather than relying solely on teacher judgment or objective test scores.

  16. What is Student Peer Review and What Does it do for Student Learning? (cont’d) Benefits of Peer Review – 1) Social Constructivism (cont’d) • The social constructivist process model of assessment argues that the peer-marking process was found to be particularly effective in improving students’ work and in students’ positive perceptions of the value of the activity when model answers were used (O’Donovan, Price and Rust, 2004). • As argued by O’Donovan, Price and Rust (2004 at p. 13), these findings “…arguably demonstrate that inviting students into the marking process can mean that assessment broadens out from merely the assessment of learning to become an effective learning tool in its own right, facilitating assessment for learning.” • In addition, the benefits of peer assessment have been highlighted as including the fact that students become more confident, independent and reflective learners, and they obtain a deepened understanding of the required learning (O’Donovan, Price and Rust, 2004).

  17. What is Student Peer Review and What Does it do for Student Learning? (cont’d) Benefits of Peer Review (cont’d) – 2) Peer Learning Networks • A second benefit of peer review is that it has the potential to assist students from culturally and educationally diverse backgrounds in adjusting to university, with peers potentially acting as positive role models within a non-intimidating, informal environment. • As highlighted in Ladyshewsky and Gardner “communications between peers are less threatening than those that involve supervisors or authorities. Hence, enhanced disclosure, discussion and deeper learning outcomes are possible”.

  18. What is Student Peer Review and What Does it do for Student Learning? (cont’d) Benefits of Peer Review – 2) Peer Learning Networks (cont’d) The advantages of these learning networks then include: • additional assistance with challenges from peers; more perspectives on problems; • access to expertise; • more meaningful participation; • the creation of an informal environment as opposed to the highly structured lectures and tutorials run by perceived authority figures.

  19. What is Student Peer Review and What Does it do for Student Learning? (cont’d) Benefits of Peer Review (cont’d) - 3) Generation of an Iterative Cycle of Learning through Formative Feedback • As highlighted by Pearce, Mulder & Baik, the level of the qualitative and quantitative feedback normally available to students involved in a major project is limited to a final summative grade from the time-poor, academic staff. • This approach is ineffective as there is no further opportunity for students to improve on their assignment. This means there is little motivation for them to reflect on, or learn from this feedback.

  20. What is Student Peer Review and What Does it do for Student Learning? (cont’d) Benefits of Peer Review - 3) Generation of an Iterative Cycle of Learning through Formative Feedback (cont’d) Thus, a third key benefit of the peer review process is its simultaneous ability to reduce the marking loads of staff while creating opportunities for students to become involved in a continuous cycle of evaluating the work of their peers during its formative stages. The benefits of this evaluation process then include that it: provides students with a valuable perspective on their own work; encourages them to revise it; promotes a sense of community and collaboration; and helps students to become equipped for lifelong, independent learning.

  21. Limitations of Peer Review Student Concerns: • While the available literature highlights a wide range of benefits of peer review, there are a range of potential impediments to implementing student peer review including: • with students rather than staff marking the work, issues of validity, reliability, bias and fairness will arise; • students may dislike evaluating another student’s work;

  22. Limitations of Peer Review (cont’d) Student Concerns (cont’d) • students can resent being required to review and comment on other students’ work believing that staff are paid to complete these tasks; • they may lack confidence in their own ability to evaluate their peers’ work and may similarly doubt the competence of other student reviewers; and • and some students may, on a cost versus benefit analysis, feel that the time taken to provide a peer review is not compensated for by the comments received by the peer review they will receive in exchange.

  23. Limitations of Peer Review (cont’d) Staff Concerns: • Of concern to academic staff is that any teaching innovation must be achieved within short time frames to ensure minimal impact on their key disciplined-based research and publication agendas. • Thus, the key issue for staff in terms of the adoption of any peer review process related to the time needed to administer the peer review process in class on a manual basis.

  24. Minimising Peer Review Limitations: The Introduction of E-Learning Technology (Case Study Two)and Reflective Practices (Case Study Three) • As highlighted in the on-line PRAZE support manual by Pearce, Mulder & Baik (2007), their development of an e-Learning based, peer review process at the University of Melbourne (PRAZE), a Web-based system that facilitates automation of anonymous peer review amongst students, was motivated by the desire to provide students with feedback that promotes a genuinely reflective cycle of learning.

  25. Minimising Peer Review Limitations: The Introduction of E-Learning Technology (Case Study Two)and Reflective Practices (Case Study Three)– (cont’d) • “...Students benefit both by being the recipient of comments on their own work but also through critically reviewing the work of others and reflecting on its positive and negative aspects. PRAZE has many similarities to systems used to assist in managing the reviewing of papers for a journal or conference, but it also has specific requirements unique to the teaching environment.” • This focus on a formative-based, reflective cycle of learning within the PRAZE peer review process, was designed to overcome the previously highlighted issue of time-poor academic staff providing only a summative grade on the final version of a submitted assignment.

  26. Minimising Peer Review Limitations: The Introduction of E-Learning Technology (Case Study Two)and Reflective Practices (Case Study Three) - (cont’d) • These costs concerns have been recognised by Pearce, Mulder & Baik (2007) who state that (at pp. 1-2): “...given the pedagogical merits of formative peer review are so well established, it is perhaps surprising that student peer review is not a more pervasive feature of university curricula. One reason is that administrating anonymous peer review without the aid of custom-designed software is so onerous that it remains a potent disincentive to implementation, especially when classes are large. Online tools promise to significantly reduce this burden, and are therefore an important part of the peer review landscape.”

  27. Minimising Peer Review Limitations: The Introduction of E-Learning Technology (Case Study Two)and Reflective Practices (Case Study Three) - (cont’d) • Student concerns related to uncertainty as to whether they have the skills and experience to mark the work of their peers are also potentially minimised given the highly structured, step-by-step marking guide that can be included within the online review process. • By providing an efficient and easy-to-use online tool for time-poor students, the costs related to any peer review completed are also minimised.

  28. Minimising Peer Review Limitations: The Introduction of E-Learning Technology (Case Study Two)and Reflective Practices (Case Study Three) - (cont’d) • However, an important consideration in the adoption of online tools is the potential for the technology to ultimately fail to deliver one of the core principles or foundations of the peer review process and that is to actively engage students in a collaborative process as equal members of a community of scholars. • Thus, while the time “costs” of staff and students can be reduced by e-learning technology, of concern to Pearce, Mulder and Baik [20] and to all academics seeking to achieve key teaching and learning objectives, is where does that leave the social benefits of a sense of belonging to a university community, which are mostly acquired through face-to-face contact?

  29. Minimising Peer Review Limitations: The Introduction of E-Learning Technology (Case Study Two)and Reflective Practices (Case Study Three) – (cont’d) • Is it possible to create a virtual sense of belonging and is this online technology equally as effective? • Case Study Two as summarised below (and as detailed by Taylor (2011 (a) and (b), highlights the key features of an independent trial of the PRAZE process and its ability to engage students in a sense of online community.

  30. Case Studies: Methodology and Results Case Study One: Financial Accounting (2010) • As highlighted by Taylor [2011 (a) and (b)], the key issues that prompted the introduction of a staff administered, peer review-based assessment task in Semester One, 2010 were low participation and attendance numbers and the high failure rates associated with the predominantly exam based assessment tasks. • Within this context, in Case Study One, a peer-review based task was introduced in the first “serious” accounting subject offered as part of an undergraduate degree. The positive outcomes achieved, as set out in the table below extracted from Taylor (2011 (a), included: failure rates declined 15%; tutorial participation increased fourfold; tutorial engagement increased six-fold; and there was a 100% student-based approval rating for the retention of the assessment task.

  31. Case Study OneStudent Results - Peer Review – Staff Administered – Semester One 2010 - AYB200 – Final Exam Results – Pre and Post Assessment Task – Extracted from Taylor (2011a)

  32. Methodology and Results (cont’d) Case Study One: Financial Accounting (2010) (cont’d) • Threatening the survival of this process however were staff issues relating to lost research time due to administrative requirements. • A resolution to this ‘time lost’ issue for staff was then sought in the remodelled, e-learning-based, peer review process as set out in Case Two below.

  33. Methodology and Results (cont’d) Case Study Two: International Accounting (2012) PRAZE Trial – E-Learning Process • As again detailed by Taylor [2011 (a)] , in order to resolve the ‘research time lost’ issues raised by full time academics as highlighted in Case One, the School of Accountancy, QUT Business School, become a trial member of the University of Melbourne’s PRAZE e-Learning project. • Within this e-learning project, in Week Seven of Semester Two, 2011, the students in International Accounting were asked to submit/upload their draft only and to do this anonymously, i.e. by student number.

  34. Methodology and Results (cont’d) Case Study Two: International Accounting (2012) PRAZE Trial – E-Learning Process (cont’d) • In addition, a step-by-step pro-forma of review questions had been set up within the PRAZE system which guided the students through their review of the peer task assigned to them and which utilised the major project’s primary assessment criteria. • The students were provided with a four-day submission phase and then a four-day review phase to allow for students who were ill, away on work-related tasks, or who had other assessment tasks deadlines to meet.

  35. Methodology and Results (cont’d) Case Study Two: International Accounting (2012) PRAZE Trial – E-Learning Process (cont’d) • Of the 126 students in this (off-semester) cohort, 102 students submitted their drafts and then 99 students of these 102 students completed their assigned reviews. • The submission phase was assigned 4%, while the on-line peer review was assigned 12 marks within the overall project assessment total of 90 marks. • The overwhelmingly positive results of the voluntary and confidential  survey process undertaken in the Week Thirteen revision lecture for this cohort of students, with 92 students completing the survey, are set out in Fig. 1.

  36. THE RESULTS - Fig. 1 – CASE STUDY TWO – E-Learning Administered – AYB227 – 92 Peer Review Student Evaluations – Voluntary and Anonymous PRAZE Submission and Review Process

  37. THE RESULTS - Fig. 1 – CASE STUDY TWO – E-Learning Administered – AYB227 – 92 Peer Review Student Evaluations – Voluntary and Anonymous PRAZE Submission and Review Process (cont’d) The key positive issues highlighted in the Fig. 1 summary of the 92 students who participated in this survey and who responded at the Agree or Strongly Agree level were that: 1) the peer review task submission deadlines and marks allocated provided an excellent motivator to start the assignment early (87/92); 2) & 3) both the PRAZE-related submission and review procedures were very easy to use (88/92 and 86/92 respectively); 4) that the peer review process assisted the students to more fully understand what was expected of them in order to complete the set task (72/92

  38. THE RESULTS - Fig. 1 – CASE STUDY TWO – E-Learning Administered – AYB227 – 92 Peer Review Student Evaluations – Voluntary and Anonymous PRAZE Submission and Review Process (cont’d) 5) the quality of the peer review comments received were of great value to the students (74/92); 6) the marks awarded for participation in the peer review process was felt to be very fair (85/92); and 7) that the peer review process should definitely be retained, and, if possible expanded to multiple reviews per student  (85/92).

  39. THE RESULTS - Fig. 1 – CASE STUDY TWO – E-Learning Administered – AYB227 – 92 Peer Review Student Evaluations – Voluntary and Anonymous PRAZE Submission and Review Process (cont’d) • Of concern, however, were a range of verbal comments received from students who felt that they needed further guidance on how to more effectively review the work of their peers and in terms of how to best reflect on and then process the feedback they had received from their reviewers. • These issues were then the subject of a special survey which was completed with the International Accounting cohort of students in Semester Two, 2012 as detailed in Case Study Three in the following section.

  40. Methodology and Results (cont’d) Case Study Three – International Accounting, 2013: Introduction of Reflective Practices • In order to more fully explore the feedback issues of concern raised by the students within the context of Case Study 2, the Semester Two, 2012 cohort of International Accounting students were surveyed on a voluntary and anonymous basis. The key results from this survey are highlighted in Figs. 2 and 3. • The initial response to the question of “how did you approach the peer review process?”, was very positive with 94% (34/36) of the students who participated in the survey being ‘open to reviewer feedback’ as set out in Fig. 2.

  41. Fig. 2: CASE STUDY THREE - Student Survey Results - Peer Review – Initial Approach to Peer Review Process – Semester Two, 2012

  42. Fig. 3: CASE STUDY THREE- Process for Dealing with Review Feedback - 36/42 Students Surveyed Participated in Peer Review Process - Multiple Responses per Student

  43. Methodology and Results (cont’d) Case Study Three – International Accounting, 2013: Introduction of Reflective Practices (cont’d) • However, in responding to the survey question of ‘what was your process for dealing with the reviews received?’, Fig. 3 clearly highlights that students did indeed have concerns with some aspects of the peer review process as it had been formulated within the Case Two context. • For example, there were significant issues of ‘mistrust’ and ‘conflict’ (15/36 - 42%) in relation to how to best reflect on and process the feedback received as highlighted in Fig. 3. In order to resolve these issues 33% (12/36) of students sought advice from other students and staff prior to accepting their reviewer comments.

  44. Methodology and Results (cont’d) Case Study Three – International Accounting, 2013: Introduction of Reflective Practices (cont’d) • In addition, Fig. 3 also indicates that 28% (10/36) of students felt both ‘confused’ and ‘hesitant’ to move away from their own ideas and found themselves continually ‘justifying their original arguments’ against what they perceived were the ‘attacks’ of the reviewers. • In seeking then to address these feedback-related issues of concern, a reflective approach was considered to offer a way forward in developing students’ evaluative and transformative learning skills.

  45. Methodology and Results (cont’d) • Case Study Three – International Accounting, 2013: Introduction of Reflective Practices (cont’d) • As highlighted by Ryan & Ryan, the importance of reflection in higher education and across disciplinary fields is widely recognised and it is generally included in university graduate attributes, professional standards and programme objectives. • However, a key issue is that reflection is commonly embedded into assessment requirements in higher education subjects, without the necessary scaffolding or setting out of clear expectations for students.

  46. Methodology and Results (cont’d) • Case Study Three – International Accounting, 2013: Introduction of Reflective Practices (cont’d) • Also of concern is that researchers and commentators agree that there are different types or hierarchical levels of reflection that need to be taken into account when designing a peer review task. • For example, Bain et al. (2002) suggest different levels of reflection with their 4Rs framework of Reporting and Responding, Relating, Reasoning and Reconstructing as detailed below in Fig. 4.

  47. Fig. 4: CASE STUDY THREE – Extracted from Bain et. al. [4] – the 4Rs Framework – Reporting and Responding, Relating, Reasoning and Reconstructing

  48. Methodology and Results (cont’d) Case Study Three – International Accounting, 2013: Introduction of Reflective Practices (cont’d) • Also of concern is that, in spite of the rhetoric around the importance of reflection for ongoing learning, there is scant literature on any systematic, developmental approach to teaching reflective learning across higher education programmes/courses.

  49. Methodology and Results (cont’d) Case Study Three – International Accounting, 2013: Introduction of Reflective Practices (cont’d) • To overcome this gap in the literature and teaching practice, Ryan & Ryan [2012 (a)] have developed ‘... a new, transferable and customisable model for teaching and assessing reflective learning across higher education, which foregrounds and explains the pedagogic field of higher education as a multidimensional space. We argue that explicit and strategic pedagogic intervention, supported by dynamic resources, is necessary for successful, broad-scale approaches to reflection in higher education.’ • In order to ensure that the International Accounting students receive the level of support needed to develop appropriate reflective habits, Ryan & Ryan’s teaching and assessing reflective learning (TARL) model, as detailed in Fig. 5 below, will be utilised to reformulate the e-learning based, peer-review task for application in Semester One, 2013.

  50. Fig. 5: CASE STUDY THREE - The TARL Model - Ryan & Ryan [2012 (a)] - applying adapted levels from Bain et. al. (2002)

More Related