1 / 26

Student readiness

Student readiness. Supervision challenges. Towards a Thesis Assessment Matrix: An action research project Michelle Picard & Lalitha Velautham. Challenges of format. New examiner challenges. Literature on thesis examination.

heman
Télécharger la présentation

Student readiness

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Student readiness Supervision challenges Towards a Thesis Assessment Matrix: An action research project Michelle Picard & LalithaVelautham Challenges of format New examiner challenges

  2. Literature on thesis examination • ‘There seems to be a tacit understanding of what constitutes a well-prepared Ph.D. student’, and that ‘in the complete absence any central repository of rules or a cosmic accrediting agency’ there is ‘extraordinary stability’ (Kwiram, 2006 p. 142 cited in Holbrook et al, 2008). • Experienced examiners generally give feedback consistently. Inexperienced examiners find it difficult to define acceptable thesis standards and are inconsistent in giving feedback (Kiley & Mullins, 2004; Kiley et al, 2011). • Need for indicators to distinguish between theses of threshold quality and those of highest quality (Holbrook et al, 2008) • The global research/ the global thesis standard? RED, School of Education

  3. Literature on assessment matrices • Tools to negotiate expectations between supervisors and students (Cadman & Cargill, 2007, Kumar & Strache, 2007) • Taxonomies for reviewing literature (Cooper, 1988; Boote and Beile, 2005; Randolph, 2009) • Thesis rating scale (Albertyn et al, 2011)

  4. Areas for extension • No clear empirical basis • Lack descriptive language and quality indicators commonly employed by experienced examiners • Disciplinarily slanted towards Sciences • Focussed on traditional thesis formats

  5. Research aim • To create thesis assessment discussion tools for supervisors, students and (potentially) examiners in order to clarify expectations and assist in assessing research documents across formats and disciplines RED, School of Education

  6. Theoretical Framework and Methods Identification of issue Development of theory Development of matrices Deployment & evaluation of matrices (Drummond & Themessl-Huber 2007) RED, School of Education

  7. Action research spiral 1 1. Research proposal assessment matrix • Developed and refined based on discussions with EAL students and their supervisors • Final product developed from RSD7 & Literature Review (Willison, 2008 & Boote & Beile 2011) • Used with over 600 research students from March 2010 and their supervisors as: • Negotiation tool • Assessment tool • Overwhelmingly positive evaluation • Wordiness RED, School of Education

  8. Action research spiral 2 2. First draft thesis assessment matrix • Imperatives from our experience & the literature as well as work on Research Proposal Matrix • Draft matrix developed from RSD7 & Literature Review (e.g. Willison, 2009; Randolph, 2009; & Boote & Beile 2011) as well as insights from spiral 1. • Deployment to experienced supervisors at the University of Adelaide via an online survey, by students in a thesis writing group and at a postgraduate research conference in South Africa (April, 2013). RED, School of Education

  9. Student comments Spiral 2 HDR students • ‘It is good to have something to take in to discuss my writing with my supervisor’ (HDR 15). • ‘It helps me to know what I must do to do research in a PhD’ • ‘It is scary to see how much I still have to learn, I thought I knew all about research in honours’ (HDR1 • ‘It is a relief to see I am ok, at a stage of development just like my supervisor’ (HDR 21) • ‘This grid really helped us to see what is expected of a thesis...you know you say it is not a Nobel Prize, but sometimes my supervisors seem to expect that... so it helps to know what a thesis really should do’. • Abit ‘wordy’ (HDRs 1,12 & 16)

  10. Conference comments • ‘Too focussed on literature review and writing skills’ • ‘Needs more focus on original contribution’ • ‘Good to consider thesis by publication’ • ‘Likely to be less useful in creative disciplines’ • ‘Does not really address argument-based thesis’ • ‘It still does not encapsulate the true work of the PhD’

  11. Experienced supervisors

  12. Positive responses in…. • Clarify expectations for research students • Clarifying expectations for new supervisors • Clarifying expectations for new examiners (but also high strongly disagree) • Aid in researcher training

  13. Positive comments • ...this would be very helpful for examiners - provides clear differentiation and progression, and what is suitable for each level.(ID 12) • ...any template which gives students, supervisors, etc. an understanding of levels of competency will assist them to distinguish between competent and excellent work. (ID 12) • the criteria listed coincide with the long established measures of adequacy in our discipline. (ID 17)

  14. Less favourable responses in… • Assist new examiners (22.2 strongly disagree) • Other thesis formats (27.8 disagree) • Differentiate between MPhil, Doctorate, PhD (33.3 neutral)

  15. Examiners & Supervisors • ...it introduces thresholds that do not correspond with the Pass/Fail approach to research degree examination. There is absolutely no point in differentiating between 'Pass' and 'Excellence' at the PhD level. (ID 10) • New students, supervisors and even examiners may still lack the knowledge or confidence, even with these criteria, to work out where their work sits... (ID 12) • If a supervisor doesn't know what is entailed in a thesis of particular kind at a particular level, he/she should not be a supervisor. (ID 16)

  16. Formats & disciplines • Please note that creative theses (creative writing PhD and MPhil) would need an adapted assessment tool - to be developed by the relevant discipline(s). One size does not fill all.(ID 3) • fails to address metrics related to thesis writing specifically in the Sciences related to areas of poor experimental design, data analysis, and data interpretation

  17. Differentiation • ‘It wasn't clear to me that the draft assessment tool provided these metrics, but rather was couching inadequacies within more general and rather heavily semantic terminologies - the RSD has its own lingua franca that may not be widely understood or appreciated, and certainly I'd suggest is beyond the interpretation of a busy international examiner. (ID 8) • ‘Experienced examiners will dismiss it. Inexperienced examiners will be misled by it.’ (ID 10) • ‘...it is implicitly language rich, and with 50% of our students coming to UA from o/seas, to introduce them to such refinement of language early in their research careers, is surely asking a bit much.(ID 8)

  18. Differentiation • If it is to have 'currency' it would need to have greater clarity through simplicity. The essential feature for a PhD, for example, is originality. This is the crucial ingredient. For the MPhil, however, the originality looms less large. (ID 10) • ‘excellent research outcomes’ often overshadows ‘poor thesis construction’. (ID 8) • A PhD degree is examined on a Pass/Fail basis. There is no recognised international concept of a 'Pass' PhD as opposed to a 'Better than Pass' PhD. (ID 10)

  19. Action research spiral 3 3. Refinement of thesis assessment matrix • Audit of rubrics provided to Australian and international thesis examiners by institutions • An audit of recurring phrases indicating quality and rigor in examiners reports • Combination of all data into revised thesis assessment matrix • Thus far, audit of rubrics completed RED, School of Education

  20. Categories of phrases

  21. Rubrics Across GO8 Universities • Knowledge contribution PhD “Thesis as a whole is substantial and original contribution to knowledge of the subject…” (ANU; UWA) “Significant contribution to knowledge and understanding in the relevant field” (MU) “Makes a distinct contribution to knowledge” (UOS) “Thesis displays new perspectives and/or advanced knowledge…” (UNSW) “makes a distinct contribution to knowledge because of the originality of the approach and/or interpretation of the findings and in some cases the discovery of new facts” (UM) University of Adelaide

  22. Rubrics Across GO8 • Knowledge contribution Masters “Shows a sound knowledge of the subject…evidence of some independence of thought” (UWA) “Makes a contribution…” (MU) “Appreciation of relationship of topic to wider field of knowledge” (UOS) “Displays new perspectives” (UNSW) “Demonstrates advanced learning in research skills” (UM)

  23. Potential matrix advantages • Supervisors: Clear articulation of expectations and standards • Examiners: Fair, transparent and consistent evaluation, assistance in articulating concerns • Students: Clear goals and direction leading to better self-management techniques in thesis writing RED, School of Education

  24. A word of caution • Hijacking by rampant managerialists/ administrators • Difficulties of pinning down the ephemeral • Danger of ‘marking’ rather than ‘examining’ (Ward, 2013). • Matrix/grid implies grades – the PhD is not graded per se • Disciplinary peer review element of becoming ‘part of the club’, entering into an in-depth discussion could potentially be lost • Personally now with two PhD examinations – still need for assessment and comments, but helped in examining process

  25. Next step in Action Research Cycle • Audit of reoccurring phrases/ language used by experienced examiners • Refining current matrix (in handout) • Trial of current matrix with examiner volunteers

  26. Questions & Feedback • Please contact: michelle.picard@adelaide.edu.au Or lalitha.velautham@adelaide.edu.au RED, School of Education

More Related