QUANTITATIVE INQUIRY

1. QUANTITATIVE INQUIRY Hwa Chong Institute Semester I, 2010

2. Advance Organizer • Introductions • Unit structure and outline • Textbook and reading materials • Stages of conducting research:

3. 1. Conceptualisation: The What and Why Issues in Education Research

4. 1.1. General Orientation to Education Research

5. The Purposes of Education Research • Why read education research? Notions of evidence-based practice • Why do education research? Adding to the stock of knowledge in the discipline

6. OECD Definition of Research • Creative work undertaken on a systematic basis in order to increase the stock of knowledge, including knowledge of humanity, culture and society, and the use of this stock of knowledge to devise new applications. • Any activity classified as research and experimental development is characterised by originality; it should have investigation as a primary objective and should have the potential to produce results that are sufficiently general for humanity's stock of knowledge (theoretical and/or practical) to be recognisably increased. • This can include both empirical and non-empirical scholarly work (added summary).

7. 1.2. Developing Interests into Questions

8. What is a “good” research question? • From where do you source your questions? • Previous literature • Theory • Empirical pilot data • FINER framework: • Feasible • Interesting • Novel • Ethical • Relevant

9. Class Exercise: Identify Problems with these Questions • Do females lack directional sense? • University lecturers’ attitudes towards student evaluations of teaching • Do teachers use ICTs more in the classroom now than before? • Management of children with behavioural problems in classrooms • What school problems are faced by children diagnosed with AD/HD?

10. Types of Research Questions • Empirical and non-empirical • “Quantitative” and “qualitative” questions • Some key concepts: • Ontology – concerned with articulating the nature and structure of the world • Epistemology – concerned with the nature of human knowledge • Methodology – concerned with principles and procedures of inquiry • Paradigm (interpretive framework) – net of epistemological, ontological, and methodological premises that guide research actions

11. Quantitative Research • Ontological • Existence of objective, absolute truths • Focus on operational definitions and rational explanations • Epistemological • Researcher (knower) and object of study (known) independent – focus on objectivity • Assumption that inquiry can approximate objectivity (i.e., be value-free); disagreements between observers due to errors and/or observer biases • Methodological • Replicability as a means for testing truth • Focus on generalization • Criteria – notions of internal/external validity

12. Qualitative Research • Ontological • Reality is local and specific • Constructions cannot be absolutely true or correct (but can be less sophisticated/informed) • Reality actively constructed rather than discovered • Epistemological • Researcher and object of study inherently dependent • Inquiry inherently value-bound • Multiple interpretations can be equally valid • Methodological • Focus on induction • Relative lack of emphasis on generality – use purposive samples • Criteria – trustworthiness, credibility, transferability, confirmability

13. Positioning in “the” qualitative/quantitative debate • The purist • Qualitative and quantitative methods are incompatible (grounded in different ontologic and epistemologic assumptions) • Advocate mono-method studies • The situationalist • Both approaches have merit for answering different types of research question • Advocate mono-method studies but accept the two approaches as complementary • The pragmatist • Dichotomy is false; many associations with each paradigm erroneous (e.g., experiments must be quantitative) • Advocate mixed-method approaches

14. Class Exercise: Classifying Research Questions • What is the effect of cooperative learning methods on mathematics achievement in sixth graders? • What are the perspectives of primary level teachers on using suspension as a behaviour management strategy? • What is the relationship between severe behaviour problems and academic performance? • How do English teachers deal with students with disabilities in the secondary level classroom?

15. Individual Exercise: Formulate a small set of possible questions related to an area of interest to you and note whether these lend themselves to a qualitative or quantitative approach

16. 1.2. Presenting a Rationale

17. Main Elements • Establish the importance of the question and place in a meaningful context. • Review what has already been done through a review of the literature • Present a thesis/argument and demonstrate how the questions derive from this background

18. Conducting a Systematic Review of the Literature (Example from Boakes, 2009) • 2.2. Literature Review Method • To ensure that the background for the current research programme was based on a systematic review of previous AP research, an explicit set of procedures was used in first locating this literature. Initially, all studies which made reference to investigating or using AP were collated. Inclusion/exclusion criteria were then applied systematically to determine which of these should be considered to inform the hypotheses proposed. • 2.2.1. Search Procedures • As indicated, the initial literature pool included all studies that had used or investigated AP effects. The only exclusion criterion applied at this initial point was that the articles had to be available in English. To generate the initial pool, systematic searches were conducted using the following databases: PsycINFO, Science Direct, ProQuest 5000 International, and Google Scholar. Once these studies were gathered, the references in each were scanned for other articles that might be relevant. Previous literature reviews were also consulted for any additional studies.

19. 2.2.2. Inclusion and Exclusion Criteria • A vast array of different procedures has been used in studies that have purported to examine AP effects. After generating the initial study pool, the studies were sorted and their key methodological characteristics tabulated. Inclusion/exclusion criteria were then generated based on the following categories. • Inclusion Criteria. Various study characteristics were deemed to be potentially important moderators of the AP effects obtained, but not to invalidate studies as investigations of these effects. These were: • Format of Prime-Target Pairs:Three main types of stimuli have been used in previous studies as primes and targets: pictures, facial expressions, and words (e.g., nouns, adjectives, and verbs). All combinations of these were included. • Attention to the Prime:Studies varied considerably in terms of the instructions issued to participants regarding the prime. In some cases, participants were asked to attend to the prime. In others, they were told to ignore the prime or were given no instructions relating to the prime. All such studies were included in the review. • Exclusion Criteria. Clearly, as the goal of this review was to explore the effects obtained in “basic” (i.e., not applied) AP studies, any studies which applied AP in the investigation of other processes (e.g., racial stereotyping, depression, anxiety) were excluded.

20. Multiple Purposes of a Rationalefrom http://www.meaning.ca/archives/archive/art_how_to_write_P_Wong.htm • Ensures that you are not "reinventing the wheel". • Gives credits to those who have laid the groundwork for your research. • Demonstrates your knowledge of the research problem. • Demonstrates your understanding of the theoretical and research issues related to your research question. • Shows your ability to critically evaluate relevant literature information. • Indicates your ability to integrate and synthesize the existing literature. • Provides new theoretical insights or develops a new model as the conceptual framework for your research. • Convinces your reader that your proposed research will make a significant and substantial contribution to the literature (i.e., resolving an important theoretical issue or filling a major gap in the literature).

21. Common Problems with Rationales from http://www.meaning.ca/archives/archive/art_how_to_write_P_Wong.htm • Lacking organization and structure • Lacking focus, unity and coherence • Being repetitive and verbose • Failing to cite influential papers • Failing to keep up with recent developments • Failing to critically evaluate cited papers • Citing irrelevant or trivial references • Depending too much on secondary sources

22. Small Group Exercise: Discussion of Brief Rationales

23. 2. Research Design: The How Issue in Education Research Quantitative Approaches

24. 2.1. Variables in Quantitative Research • Variable – a phenomenon that is changeable or subject to variation • Independent variable – the predictor or presumed effect variable • Dependent variable – the outcome variable • Types of variables – categorical, ordinal, interval, ratio

25. Class Exercise: Identifying and Classifying Variables

26. A. A study indicated that wall colour may reduce the time taken by seventh graders to complete • simple arithmetic tasks. • Grade level: ______________ • Time taken: _____________ • Wall colour: ________________ • B. A study indicated that attitudes toward using ICTs in the classroom differed across primary and • secondary teachers. • Attitudes toward using ICTs: ______________ • Phase of schooling (i.e., primary/secondary): ____________________ • C. A study indicated that amongst male teachers, attitudes toward quality assurance differed with • time spent in the system. • Sex of teacher: ________________ • Attitudes toward quality assurance: ____________ • Time spent in system: _________________

27. 2.2. Sampling in Quantitative Research • Probability-based sampling • Aimed to ensure that individuals have the same chance of being included in the sample • Simple random sampling procedures (e.g., random number generation) • Stratified random sampling (e.g., identify subpopulations of interest) • Non-probability-based sampling • Convenience sampling (grabbing anyone willing) • Snowball sampling (ask respondents to nominate others who might be willing to participate)

28. 2.3. Quantitative Data Collection: Measurement Issues • Common sources of quantitative data: • Archival records • Direct observation • Questionnaires, inventories, scales, tests • The notion of operational definitions • Conceptual – abstractions that facilitate understanding • Operational – statement of definition in measurable terms

29. Common domains of measurement in Quantitative Research • The cognitive domain • The affective domain • The psychomotor domain

30. Common approaches to measurement • Cognitive domain: • Selected response • Short answers • Cloze tests • Essays • Affective domain: • Questionnaires • Interviews • Psychomotor: • Performance-based assessment • Structured observation

31. Technical Adequacy • the consistency of assessment: reliability • the appropriateness of interpretations of assessment: validity • Bias in assessment • interrelationships between reliability and validity • other indexes of test characteristics

32. The Purpose of the Standards • to promote the sound and ethical use of tests • to provide assessment professionals with • guidelines for the evaluation, development, and • use of testing instruments • to provide a frame of reference for addressing relevant issues

33. Stability of Assessments: Reliability

34. Classical test theory and the true score model: • Observed score = true score + random error • Common Sources of Measurement Error • inconsistencies across testing occasions • inconsistencies across forms of the tests • inconsistencies between raters • inconsistencies in sampling the content domain

35. Key Concepts • reliability as a characteristic of test scores • major approaches to establishing reliability and links to specific error sources • statistical basis for reliability analyses: the reliability coefficient and the standard error of measurement

36. Common Approaches to Establishing Test Reliability • The Test-Retest Method • Reactivity and practicality • Alternate Forms Methods • Practicality and expense • Internal Consistency Methods • Limited information on reliability

37. Some Current Issues in Evaluating Reliability • limitations of the traditional approach • factors affecting reliability coefficients • obtaining independent estimates of error sources and interactions between error sources • Generalisability (G-)Theory

38. Purposes of Assessments: Validity

39. Key Concepts • validity as a property of test scores • validity as a matter of degree • specificity to particular uses or interpretations • Traditional Approaches to Validation • historical developments in conceptions of validity • the three-part framework: criterion, content, and construct validation

40. Content-Related Validity • The Standards Definition: • the extent to which the sample of items, tasks, or questions on a test are representative of some defined universe or domain of content • Approaches to Establishing Content-Related Validity • domain sampling, relevance, clarity • tables of specifications: content domains and structure • logical analysis of test content • examining test content and format • assessing suitability for the given purposes

41. Criterion-Related Evidence • The Standards Definition: • the extent to which scores are systematically related to one or more outcome criteria • Approaches to Establishing Criterion-Related Validity • predictive validity • concurrent validity • assessing criterion-related evidence according to decision purposes

42. Construct-Related Evidence • The Standards Definition: • extent to which test measures the identified underlying characteristic of interest • Approaches to Establishing Construct-Related Validity • explicating construct meaning • convergent and divergent evidence • deriving and testing predictions about test performance from the underlying theory

43. Small Group Exercise: Constructing an Operational Definition of Bullying

44. Defining bullying • Conceptual definition: Tattum and Tattum (1992) - "Bullying is the wilful, conscious desire to hurt another and put him/her under stress". • A. Provide an operational definition of bullying based on the conceptual definition provided.

45. B. Evaluate the following behaviours according to your operational definition: • Hitting someone for no reason • Gossiping about another person • Coercing another person to do something they don’t want to do • Taking the possessions of another person • Telling another person what to do • Refusing to speak with another person because of their personal characteristics

46. C. Evaluate the following scenarios according to your operational definition: • Jack is a slight boy in the 8th grade who seems to have “fallen out” with a dominant group of other boys in his year. At least once per week, a group of five other boys play a prank on him, which can include stealing and hiding his locker key, to more physical pranks such as removing his trousers in front of his peers. • Michael is a popular 11th grader with his peers. He is seen as a leader amongst members of his peer group. There is one boy in his year, however, who has taken a disliking to him. This boy frequently attempts to tease him over a number of minor issues (e.g., clothing, and his popularity with others). Michael takes no notice of this, and appears to “laugh it off”, because the boy who is engaging in these behaviours is himself unpopular with his peers.

47. Sandra is in the 10th grade. She is socially isolated, generally spending her recess and lunchtimes alone. When she attempts to engage with other girls from her year group in joint activities in class, they refuse to speak with her. She is never invited to social events outside school.

48. The Revised Standards Framework

49. The 1999 Revision • has more background material • greater number of standards • reflects changes in federal law and measurement trends affecting validity, etc. • addresses professional and technical issues of test development and use • Move towards sources of validity evidence rather than distinct types of validity • Notion of consequences (Messick) explicitly taken into account

50. Validity • Focus on the obligations of the test developer to users,examinees, and other testing practitioners on user obligations to examinees • Developers owe users • Enough information to make judgments about the appropriateness of their interpretation of test scores for their intended use(s) population(s) for which test is appropriate constructs tested • uses/interpretations NOT intended or ecommended • content descriptions, domains, criticality • qualifications of experts/judges/raters • rating/scoring procedures • population/situation variables involved in validation