How is e-learning quality assessed in Finland: FVU’s Referee service

1. How is e-learning quality assessed in Finland: FVU’s Referee service Petra Rutanen University of Oulu Dept. of Electrical and Information Engineering E-XCELLENCE+ seminar May 20, 2009

2. Referee service – a part of the service operations of the Finnish Virtual University • ”Peer review of the quality of online learning materials” • The goals of the service are: • To provide evaluation criteria and a service model that can be used to carry out quality assessment of online material • To give recognition for e-learning material producers (and evaluators) • To increase the common usability of high-quality online learning material produced in universities.

3. Referee service – a part of the service operations of the Finnish Virtual University • A portal site where: • Any e-learning material producer can request an evaluation for their material • Users can search for evaluated materials • Material producers can test their own material • Experts can apply to be evaluators • Guide and introduction for the evaluation process The referee service defines online learning material as an educational entity with contentual requirements and goals set for learning.The service focuses on examining online learning material that its producer wants to bring into common use and wants a referee evaluation quality mark for.

4. The referee service in refreepalvelu.fi • The producers of online learning material get • recognised evaluation of the online learning material they have produced • ideas for further development of the learning material • recognition for the quality of their work • publicity (and users for the produced learning material) • The users of online learning material get • support for choosing and using high-quality material • (ideas for developing their own teaching methods) • examples of high-quality online material • The evaluators of online learning material get • information on factors that affect the quality of online learning material, and examples of high-quality learning material • Orientation for the process • evaluation merit and a certificate.

5. The Referee project 2007-2008 • There was a need for: • quality assurance • develop an evaluation procedure that meets the challenges of the production process • give merit to e –learning material producers • The aim was to create an assessment model that is extensive but does not excessively burden the referees. • The service was developed in 2007-2008 as a joint Referee service project between the coordinator Lappeenranta University of Technology, Tampere University of Technology, University of Oulu, University of Joensuu and the FVU Service Unit.

6. The evaluation criteria for online learning material • The Referee service evaluation criteria are based on various national and international evaluation criteria for online learning material • Four categories: common usability and pedagogical, content, and instrumental criteria • E-learning material is reviewed criterion by criterion under the main named categories • In every main criterion category there are 2 to 3 criteria which are evaluated • The evaluation focuses on a total of 27 criteria; every division has an supporting description • Every criteria is evaluated with accuracy by 0,25, 0,5 or 1 point. • The maximum points are: 7 p for each main category, 28 p totally

7. Criteria for common use • 1.1 Availability • A. Stability of use • B. Accessibility • C. Description of the user rights • 1.2 Descriptional information • A. Findability of the descriptional information • B. Meta data • 1.3 Transferability • A. Usability in other teaching contexts • B. Instructions for other teachers

8. Pedagogical criteria • 2.1 Goals of learning • A. Description of the goals of learning • B. Correspondence between the goals of learning and the online material • 2.2 Target group • A. Description of the target group • B. Consideration of the target group • 2.3 Supporting learning • A. Directiveness of the learning process • B. Overall learning experience

9. Contentual criteria • 3.1 Relevance • A. Suitability of the contents • 3.2 Reliability and up-to-dateness of the factual contents • A. Author information • B. Use of information sources • C. Reliability • D. Up-to-dateness • 3.3 Clarity and diversity of the presentation method of the contents • A. Clarity of the contents • B. Diversity of the contents

10. Instrumental criteria • 4.1 Usability • A. Ease of use • B. Manageability • C. Graphical/visual functionality • 4.2 Barrier free • A. Consideration of different user environments • B. Consideration of different users in technical implementation • C. Technical requirements • D. Technical reliability

11. The Referee project • 16 e-learning materials were evaluated during 2007 • After feedback,11 materials were evaluated during 2008 • Evaluation groups consisted of Referee project actors, experts on e-Learning and each quality criteria, and also student members • Every material passed the assessment (qualification limit 10 points) • The points converted into stars (1-5 stars) • The producers of e-learning material got detailed reports of the evaluation • Feedback was collected both from evaluators and the material makers

12. www.refereepalvelu.fi • Feedback results • Evaluators: • The process was described as interesting, smooth and educational • Fuctionality of the process got a rating of 4,33 (1-5) • The use of time was suitable, the whole process took on average of 11 hours of time/evaluator • Producers: • Satisfied with comprehensive evaluating • Valuable ideas for developing the material • Development as an employee • The mean value of satisfiction with the evaluation report was 3,90.

13. Thank you! petra.rutanen@ee.oulu.fi For details: Mr. Totti Tuhkanen (SVY) at tottituh@utu.fi www.refereepalvelu.fi