Assessment and evaluation of pedagogical outcomes in the area of Networked Supported Collaborative Learning

1. Assessment and evaluation of pedagogical outcomes in the area of Networked Supported Collaborative Learning Symeon Retalis (retal@unipi.gr) University of Piraeus Department of Technology Education and Digital Systems Computer Supported Learning Engineering Lab http://www.softlab.ece.ntua.gr/research/research_projects/tell/

2. Why do we care about the effectiveness of e-learning? • Research on e-learning has been driven by what many are calling the “information revolution.” • E-learning, which was once a poor and often unwelcome stepchild within the academic community, is becoming increasingly more visible as a part of all educational levels. • The attitudes and satisfaction of students using e-learning also are characterized as generally positive. • Thomas Russell, in his recently published annotated bibliography entitled The No Significant Difference Phenomenon, lists 355 sources dating back as early as 1928 that seem to bolster these arguments [http://www.nosignificantdifference.org]. • However, a closer look at the evidence suggests a more cautious view of the effectiveness of e-learning

3. Contemporary research on the effectiveness • Research Approaches • Descriptive research involves the collection of data to answer specific questions. • Data are usually collected through questionnaires, interviews, or standardized attitude scales. An important component of descriptive research is the validation of the questionnaire in order to determine if it measures what it was developed to measure. • Typical descriptive studies are concerned with the assessment of attitudes, opinions, and conditions. • A case study is an in-depth investigation of one “learning unit.” • The researcher can use a variety of methods to gather data, however, the explanation of the unit is generally written in narrative form. • Correlational research involves collecting data in order to determine whether, and to what degree, a relationship exists between two or more quantifiable variables. • An estimate is provided of just how related two variables are. It is important to note that correlational research almost never establishes a cause-effect relationship. • One example of a correlational study might be determining the relationship (correlation) between student satisfaction with an instructor and the type of technology used. • Experimental research is the only type of research that can truly test hypotheses concerning cause- and-effect relationships. • In an experimental study, the researcher manipulates at least one independent variable and observes the effect on one or more dependent variables. In other words, the researcher determines “who gets what,” which group of subjects will get which treatment. • The groups are generally referred to as experimental and control groups.

4. Key shortcomings of original research (I) • Much of the research does not control for extraneous variables and therefore cannot show cause and effect. • Most experimental studies of e-learning are designed to measure how a specific technology—the “cause”—impacts upon some type of learning outcome or influences the attitudes of students—the “effect.” • To accurately assess this relationship, other potential “causes” must not influence the measured outcomes. If other variables influence outcomes, it is impossible to attribute “cause” to the technology being used. • The validity and reliability of the instruments used to measure student outcomes and attitudes are questionable. • An important component of good educational research relates to proper measurement of learning outcomes and/or student attitudes. In short, do the instruments—such as final examinations, quizzes, questionnaires, or attitude scales—measure what they are supposed to measure? • A well-conducted study would include the validity and reliability of the instruments so that the reader can have confidence in the results.

5. Key shortcomings of original research (II) • Many studies do not adequately control for the feelings and attitudes of the students and faculty—what the educational research refers to as “reactive effects.” • Reactive effects are a number of factors associated with the way in which a study is conducted and the feelings and attitudes of the students involved. • One reactive effect, known as the Novelty Effect, refers to increased interest, motivation, or participation on the part of students simply because they are doing something different, not better per se. • Other shortcoming : • There is a lack of a theoretical or conceptual framework • No systematic methods for collecting and analysing/interpreting data have been followed • It is not taken into consideration how the different learning styles of students relate to the use of particular technologies

6. TELL: Towards Effective network supported coLLaborative learning activities[http://www.softlab.ece.ntua.gr/research/research_projects/tell/] University of Piraeus, GR Politechnico di Milan, IT Maastricht Learning Lab, NL National Technical University of Athens, GR University of Valladolid, ES University of Patras, GR A Priory Ltd, UK

7. Aim and objectives • This project is a methodical and systematic effort • to support the understanding of the learning process that happens in networked supported collaborative learning (NSCL) environments, • to provide methods and tools to measure the effectiveness of networked supported collaborative learning activities, • to offer means for training the human actors involved (or who would like to get involved) into collaborative learning activities and • to support the design of new effective technological tools for collaborative learning.

8. Project focus • This project focuses on specifying the concept of effective network supported learning activities within a variety of contexts, and as a synergy of • instructional methods, • technology, • subject matter, • and other contextual factors which provide the conditions necessary to support learning – seen as both knowledge construction and skill acquisition.

9. Project activities • The project consortium will also exchange know-how and experiences about • the evaluation process and • tools for networked supported collaboration and interaction between the actors (students, tutors etc.) • in multiple and diverse learning environments, • in order to provide holistic conceptual evaluation frameworks and systematize the measurement of effectiveness in quantitative and qualitative approaches. • Moreover, the project partners will cooperate in order to create software system architectural frameworks that will allow network supported collaborative learning tools to interchange data with other tools and with evaluation tools.

10. Project Work flow (3) WP1: Peer reviews of evaluation studies WP2: User trials and evaluation field research on NSCL Resources about evaluation methods and tools (WP1 deliverable) WP3: Design patterns construction WP4: Interchangeability of data among NSCL tools Project Management Tutorials and Workshops Dissemination

11. Project Deliverables • This project will offer: • a set of design patterns for NSCL • Use the experience of the ELEN project [http://www.tisip.no/E-LEN] • Resources about evaluation methods and tools for network supported collaborative learning, on which an evaluation toolkit for networked supported collaborative learning will be based. • It will develop a meta-study of evaluation studies of network supported collaborative learning and document it both online as well as in a paper based handbook. • A report based on multiple evaluation studies that will happen in real educational environments (schools, universities, workplaces, etc.) using different networked supported collaborative learning systems (Blackboard Polaris, Learning Space, 3D active worlds, group based simulations, etc.) and strategies (synchronous, asynchronous, workplace collaborative learning, etc.) • A conceptual framework for the interchangeability of data among different networked supported collaborative learning systems • Tutorials and workshops • Dissemination events and resources (web site, papers, etc.)

12. Needs in evaluation • The point to understand about evaluation is that it needs to • operate within a shared framework and • recognise the different needs of the main stakeholder groups (such as families with young children and single adults and retired people; or such as teachers, learners, parents, employers and government). • Without some kind of shared framework there are no possibilities for shared understanding.

13. Evaluands • In education, we tend to find the following kinds of “things featuring” as evaluands (things to be evaluated): • A learning resource (such as a textbook, a multimedia programme, a website) • A learning tool/platform (such as a sliderule, a modeling program) • A course (a connected set of objectives, activities and resources; usually intended for a defined target group of learners and usually also involving some form of assessment to see if learning objectives have been met) • A teaching strategy (such as lecturing to the whole class or problem-based learning) • A learning environment (a connected set of resources and tools, arranged in space and usually inhabited by a known group of learners and one or more teachers) • An innovation project (a planned sequence of activities intended to create a defined output, and/or achieve some defined outcomes, within a defined envelope of time and resources) The main point to accept here is that ‘things’ out of context are hard to evaluate. We need to plan to evaluate a complex configuration of things and processes.

14. Evaluation in the domain of network supported collaborative learning • There is a small, specialized literature on this topic (see e.g. Berge & Myers, 2000; Rossman, 1999) • The main new issues we need to take into account are: • the specific educational processes and goals associated with NSCL – for example that it is concerned with collaborative not individualistic learning, that it emphasizes learning through social interaction (mediated by technology) • the foregrounding of tools and infrastructure (we usually want to know something about the qualities, advantages, etc of new tools or infrastructure) • the importance of defined and implicit ways of working, assumptions and expectations about how to collaborate over the Internet, etc

15. Findings in NSCL (i) • In general, studies of this kind tend not to reveal significant differences between groups of students involved in NSCL and groups taught in other ways. • One of the largest studies in the literature – in terms of student numbers – is that reported by Carswell et al (2000), using data from an undergraduate course in computer science. • Lewis (2002) reports another study which shows no clear advantages of NSCL over conventional forms of learning – in some assessment tests the NSCL group performed better; in others the conventional group performed better. • Lewis’s analysis led to a suggestion that it was not so much the use or otherwise of NSCL that made the difference. A more important factor was how students engaged in NSCL – a higher level of engagement leading to better learning outcomes. • Kashy et al (2000) – high positive success rate from introducing NSCL on a large (500 student) on-campus introductory physics course. • Final grades in a web-based class correlated with the number of messages read and posted by students during the semester (Wang & Newlin, 2000).

16. Findings in NCSL (ii) • Need especially to comment on the problem that • most comparative studies (NSCL vs conventional) show no significant differences; • major stakeholders want evidence of value for money with respect to their investment in NSCL. The answer may be to focus more on • there are some good signs of educational benefit – but they are in areas that involve complex problem solving, higher order thinking, serious discussion of difficult ideas, etc. • Educational improvements here won’t be picked up unless the tests used are geared to these higher order learning outcomes • there’s a serious need for studies which are theoretically and methodologically sophisticated/powerful – in the sense that you need to have a good theoretical model of the potential benefits of NSCL to know what to look for. • Good examples are studies by Jonassen & Kwon (2001), Benbunan-Fich & Hiltz (1999)

17. Findings in NCSL (iii) • [Pinelle & Gutwin, 2000] “A Review of Groupware Evaluations” • The main findings are that • almost one-third (1/3)of the groupware systems were not evaluated in any formal way, • that only about one-quarter (1/4) of the articles included evaluations in a real-world setting, • and that a wide variety of evaluation techniques are in use. • From the studies included in this survey, 41% of the articles that • included evaluations were of actual real world software implementations, but only 25% considered the software’s organizational and work impact. Their main conclusions from the review are that more attention must be paid to evaluating groupware systems and that there is room for additional evaluation techniques that are simple and low in cost.

18. On the mainstreaming of NSCL… • using the web and communicating with others online are taken for granted. • IM and SMS are no more exotic to this generation, it seems, than note-passing and talking on the telephone were to mine, and blogging is just the modern analog of keeping a personal journal.... • In short, after barely more than 30 years for existence, NSCL has become more of a practical necessity than an object of fascination and fetish.' (Herring, 2004, p33) No “novel effect” anymore! We need to examine what works and what does not in a complex e-learning environment with relation to the high order learning outcomes

19. Step forward … • Michael Patton (1996) has pointed out, there is a long history of evaluation reports having no effect on programme and project implementation, policy and funding decisions, etc. • The main reason is that insufficient attention gets paid to the interface between evaluation reporting and decision-making. • In Patton’s view, this means that there has to be close liaison between the evaluation team and the decision-makers (at whatever level they sit), • so that the evaluators can understand the key concerns of the decision-makers and the decision-makers can anticipate how the evaluators’ evidence will help them make better decisions. • This close liaison does not have to compromise the independence and objectivity of the evaluators.

20. Need for an evaluation conceptual model … The most important part of a sucessfull design is the underlying conceptual model. The hard part of design: formulating an appropriate conceptual model and then assuring that everything else be consistent with it. Donald Norman

21. Questions? http://www.softlab.ece.ntua.gr/research/research_projects/tell/ Thanks to TELL project partners for their input