Practice Guided by Research in Providing Effective Student Support Services

1. Practice Guided by Research in Providing Effective Student Support Services Prof. Terry Anderson Canada Research Chair in Distance Education

2. Presentation Overview • Traditional Opening Joke • Need for distance education research • Sorry state of current research • Methodologies • Quantitative • Qualitative • Design-based • Dissemination and building a research culture

3. Why is Distance Education Better than Sex? • If you get tired, you can stop, save your place and pick up where you left off. • You can finish early without feeling guilty. • You can get rid of any viruses you catch with a $50 program from McAfee • With a little coffee you can do it all night. • You don’t usually get divorced if your spouse interrupts you in the middle of it. • And If you're not sure what you are doing, you can always ask your tutor.

4. “Learner support services” is the variety of non-academic interactions that the student has with a college or university, including: pre-enrollment services (recruiting, promotion, orientation), admissions and registration, academic advising, program planning, degree and transcript audit, technical assistance, library and bookstore services, personal and career counseling, social support services, and financial planning and management. (Dirr, 1999) Open University: “all those elements capable of responding to a known learner or group of learners, before, during and after the learning process” (Thorpe, 2001) Includes teaching function Defining Learner Support

5. Defining Research • “the systematic study of materials and sources in order to establish facts and reach new conclusions” (Oxford Compact Dictionary, 1991). • adjectives describing research as • disciplined, organized, • transparent, problem orientated, • public, creative, • scientific, systematic, • diligent, labourous • and accessible

6. Qualities of Research • Characterized by: • clear goals; • adequate preparation; • appropriate methods; • significant results; • effective presentation and • reflective critique. Glassick, Huber & Maeroff. (1997). Are such descriptions exclusionary?

7. Why Do Research on Learner Services? • Many unresolved questions of traditional distance education • - attrition; F2F tutorial value, paced vs unpaced; • new forms of distance education provision. • what combinations of group based learning are worth the cost and inconvenience? • Do face to face tutorials really make a difference or is real time video or audio conferencing just as effective? • How much does expensive multimedia really enhance student learning? • how important are real time interactions compared to asynchronous ones ?

8. What Makes Educational Research Effective? • Can you think of one research result that you have learned about during the past two days at this conference that will make a difference to your practice? • Why does this particular research result make a difference? • Question ; context; methodology; clarity of presentation?

9. Why Educational Research‘Just Don’t Get No Respect’ • Much research is not valued by either funders, other academics or worse, practitioners • Not funded financially • Education 0.01 % of expenditures • Health 3.0 % • High tech companies 10.0 - 15.0 % • Overall (Canada, 2002) 1.88%

10. Put in Perspective • International pharmacy giant Pfizer Inc. spent over $200 million (1999) in research related to treatments for animals – • 7 times as much as the US government spends on educational research

11. Assessment of DE Research • Many experimental research projects do not display rigour in their design • Many generalize inappropriately • Cultural, linguistic and environmental factors often not taken into consideration • Few concerned with teacher and tutor support • Few studies based on current learning, pedagogical or psychological theories • Olugbemiro Jegede (1999)

12. Barriers to Educational Research • It’s nobody’s job • How many in this room have at least 50% of their job requirement to do research? • Negligible industry support • No large scale focus on particular problems • Nobody keeping score in meaningful terms • Pervasive lack of trust in research efficacy • In sum, lack of an effective research culture • (Burkhardt and Schoenfeld, 2003).

13. How do we Build a Culture of Research in Learner Services?

14. Research Paradigms • Quantitative – discovery of the laws that govern behaviour • Qualitative – understandings from the inside • Design-based – interventions, interactions and their effect

15. Quantitative Research • From Keller to evidence based practice • Developed from health research notably Cochrane • employs a scientific discourse derived from the epistemologies of positivism and realism.

16. “those who are seeking the strict way of truth should not trouble themselves about any object concerning which they cannot have a certainty equal to arithmetic or geometrical demonstration” • (Rene Descartes,). • Inordinate support and faith in randomized controlled studies

17. Quantitative 1 – CMC Content Analysis • Anderson, Garrison, Rourke 1997-2003 • www.atl.ualberta.ca/cmc - 9 papers reviewing results focusing on reliable , quantitative analysis • Identified ways to measure teaching, social and cognitive ‘presence’ • Most reliable methods are beyond current time constraints of busy teachers • Maybe basic research as grounding for AI methods of the future

18. Is meta analysis the gold standard? • Two recent Canadian examples

19. Quantitative 2 • Bernard, et.al. (2003) How Does Distance Education Compare to Classroom Instruction? A Meta-Analysis of the Empirical Literature Concordia University • Results of 2,262 studies examined from 1985-2003, only 232 met comparative quality criteria

22. Quantitative 3 – Meta-Analysis • Ungerleider and Burns (2003) • Systematic review of effectiveness and efficiency of ICT • The type of interventions studied were extraordinary diverse –only criteria was a comparison group • “Only 10 of the 25 studies included in the in-depth review were not seriously flawed, a sobering statistic given the constraints that went into selecting them for the review.”

24. Ungerleider, C., & Burns, T. (2003) . A systematic review of the effectiveness and efficiency of networked ICT in education . P.38 Ottawa: Industry Canada. Retireved Jan. 24, 2004 from http://www.lnt.ca/technology/ict/SystematicReview.pdf

25. Quantitative Summary • Can be useful especially when testing well established practice • The need to “control” context often makes results of little value to practicing professionals • In times of rapid change too early quantitative testing may mask beneficial positive capacity • Will we ever be able to afford blind reviewed, random assignment studies?

26. Qualitative Paradigm • Many different varieties • Generally answer the question ‘why’ rather then ‘what’ or ‘how much’? • Presents special challenges in DE due to distance between participants and researchers • Currently most common type of DE research (Rourke and Szabo, 2002)

27. 1st Qualitative Example • Dearnley (2003) Student support in open learning: Sustaining the Process • Practicing Nurses, weekly F2F tutorial sessions • Phenomenological study using grounded theory discourse

28. Dearnley (2003)

29. “Support structures to facilitate personal and professional development within this context need to be in place and attention must be given to the provision of effective learner support.” (Dearnley, 2003) • “

30. 2nd Qualitative Example • Cain; Marrara; Pitre (2003) Support services that matter. • Interviews with 8 grad students • most students were not likely to take advantage of student support services. • students perceived their peers as important sources of academic and social support. • students expected their instructor to be a support resource and to be knowledgeable about the on-campus academic and administrative services.

31. Qualitative Summary • Measure of quality is “critical appraisal concerning plausibility, internal consistency and fit to prevailing wisdom • Burkhardt & Schoenfeld (2003 • But what if the only producers and consumers are researchers not practitioners? • Often the only answer that makes it to the trenches is “it depends”

32. Do Either Qualitative or Quantitative Methods Meet Real Needs of Practicing Distance Educators?

33. Quantitative vs. QualitativeParadigm Wars Rekindled • Current research “more resembles the pendulum swings characteristic of art or fashion, rather than the progressive improvements characteristic of science and technology” (p. 16). • Slavin (2002) in Educational Researcher • Solution to embrace “evidence based learning” • Projected to increase from 5% to 75% of US Gov. funding by 2007 for “research that addresses causal questions and uses random assignments …. As a result funding is likely to substantially increase” Slavin, 2002 p. 15

34. USA Department of Education (2003) guidelines for Identifying and Implementing Educational Practices Supported By Rigorous Evidence:

35. Push for “Scientifically Based” Educational Research • Scientifically based research: • employees systematic empirical methods • involves rigorous data analysis • Relies on measurements or observable methods that provide reliable and valid data across evaluators and observers • Is evaluated using experimental or quasi-experimental designs (Eisenhart and Towne 2003)

36. Push for “Scientifically Based” Educational Research • "...means research that involves the application of rigorous, systematic, and objective procedures to obtain reliable and valid knowledge relevant to education activities and programs" • (No Child Left Behind (2001)

37. Quantitative vs. QualitativeParadigm Wars Rekindled • Dualism marginalizes; stereotypes, separates and backgrounds competing positions • We have “been seduced into accepting the inherent value and power differentials that operate in the dualist construction of the 'quantitative/qualitative' binary pair of terms” (Lines, 2001)

38. Is evidence based practice better than opinion based practice? • No compelling evidence of such. • Implementation issues are much more convoluted and socially determined than the mere presentation of evidence. • Need to broaden the evidence base and allow results from diverse methodologies to enter the accumulated knowledge base.

39. But what type of research has most effect on practice? • Kennedy (1999) - teachers rate relevance and value of results from each of major paradigms. • No consistent results – teachers are not a homogeneous group of consumers but they do find research of value • “The studies that teachers found to be most persuasive, most relevant, and most influential to their thinking were all studies that addressed the relationship between teaching and learning.”

40. But what type of research has most effect on Practice? • “The findings from this study cast doubt on virtually every argument for the superiority of any particular research genre, whether the criterion for superiority is persuasiveness, relevance, or ability to influence practitioners’ thinking.” Kennedy, (1999)

41. “In any dispute the intensity of feeling is inversely proportional to the value of the stakes at issue -- that is why academic politics are so bitter.” Wallace S. Sayre, 1905-1972 quoted in, "Issawi's Laws of Social Motion" (1973)

42. 3rd ParadigmDesign-Based Research • Related to engineering and architectural research • Focuses on the design, construction, implementation and adoption of a learning initiative • Related to ‘Development Research’ • Closest educators have to a “home grown” research methodology

43. Design Studies • iterative, • process focused, • interventionist, • collaborative, • multileveled, • utility oriented, • theory driven and generative • (Shavelson et al, 2003)

44. Integrative Learning Design(Bannan-Ritland, 2003)

45. Call Centres Innovation Design-Based Research Case Study • Building Airplanes in the Air – EDS TV ad

46. Context: self paced study; 12,000 students in Business Undergraduate programs Prior to 1995 traditional telephone/email and mail tutor model

47. Tutor Model On call 2 hours/week Part time, problem with knowledge of institution knowledge of a single course Personal relationship Subject matter expertise Personal knowledge base Call Centre Model Advisor on call 40 hours/week Full time, steeped in university environment All business curriculum Customer relationship Refers to academics for subject matter expertise Formal FAQ and data collection

48. Call Centres: Answer 80% of student inquiries Saves over $100,000 /year

49. Stage 1: Informed Exploration • Review of call centre literature • Interviews with current tutors and managers • Data collection on current processes and costs • Visit to other call centres, especially those in related but uncompetitive contexts

50. Stage 2: Enactment • Design and coding using Lotus Notes • Project management, data collection on development problems and costs, • Pilot testing • Multiple iterations