THE DEATH OF MIXED METHODS? Stephen Gorard University of Birmingham s.gorard@bham.ac.uk http://www.education.bham.ac.uk/

1. THE DEATH OF MIXED METHODS? Stephen Gorard University of Birmingham s.gorard@bham.ac.uk http://www.education.bham.ac.uk/staff/gorard_stephen.shtml http://www.tlrp.org/capacity/rcbn.html http://trials-pp.co.uk

2. Frequencies of test results and disease for 1,000 people

3. Using Bayes’ Theorem p(A) ∙ p(B|A) p(A) ∙ p(B|A) + p(A’) ∙ p(B|A’) p(A|B) = In the disease example, the probabilities are: A=1%, B|A=90%, A’=99%, B|A’=10% Therefore: p(A|B) = 90 / (90 + 990), or 8%

4. Learning from experience (rather than deciding on a hypothesis) Imagine synthesising a prior likelihood of 70% (A), and then encountering a subsequent 50% conditional probability (B|A). The new posterior likelihood (A|B) is: 0.7 x 0.5 0.7 x 0.5 + 0.3 x 0.5 Still 70%, of course.

5. Learning from experience II Or, given the same prior likelihood of 70% (A), imagine encountering a subsequent 10% conditional probability (B|A). The new posterior likelihood (A|B) is: 0.7 x 0.9 0.7 x 0.9 + 0.3 x 0.1 Which is around 95%. So new evidence has merely modified the prior evidence

6. Learning from experience III Or, given a lower prior likelihood of 10% (A), imagine encountering the same subsequent 10% conditional probability (B|A). The new posterior likelihood (A|B) is: 0.1 x 0.9 0.1 x 0.9 + 0.9 x 0.1 Which is 50%. So new, unlikely, evidence has increased the prior likelihood

7. Outline of a full cycle of research

8. Conclusions 1. A key ethical concern for those conducting or using publicly-funded research ought to be the quality of the research, and so the security of the conclusions drawn. 2. It is helpful to consider the research enterprise as a cycle of complementary phases and activities, because this illustrates how all methods can have an appropriate place in the full cycle of research. 3. Working towards an experimental design can be an important part of any research enterprise, even where an experiment is not envisaged or even possible. 4. Part of the problem of research quality lies in traditional research methods training and ‘experts’. 5. Part of the problem of research quality lies in a lack of appropriate use of numbers. 6. Part of the problem of research quality lies in an unwillingness to test our cherished theories. 7. Much of the solution lies in greater scepticism, because the problem is not really one of methods at all.

9. Some references Gorard, S. (2006) Using everyday numbers effectively in research: Not a book about statistics, London: Continuum Gorard, S. (2006) Towards a judgement-based statistical analysis, British Journal ofSociology of Education, 27, 1, 67-80 Gorard, S. (2005) Revisiting a 90-year-old debate: the advantages of the mean deviation, TheBritish Journal of Educational Studies, 53, 4, 417-430 Gorard, S. (2008) Research impact is not always a good thing: a re-consideration of rates of ‘social mobility’ in Britain, British Journal of Sociology of Education, 29, 3, 317- 324 Gorard, S., with Taylor, C. (2004) Combining methods in educational and social research, London: Open University Press Gorard, S., Roberts, K. and Taylor, C. (2004) What kind of creature is a design experiment?, British Educational Research Journal, 30, 4, 575-590 Gorard, S. (2004) Scepticism or clericalism? Theory as a barrier to combining methods, Journal of Educational Enquiry, 5, 1, 1-21 Gorard, S. (2004) Three abuses of ‘theory’: an engagement with Nash, Journal of Educational Enquiry, 5, 2, 19-29 Gorard, S. and Cook, T. (2007) Where does good evidence come from?, International Journal of Research and Method in Education, 30, 3, 307-323