Research Methods of Applied Linguistics and Statistics

1. Research Methods of Applied Linguistics and Statistics By Qin Xiaoqing (Huazhong University of Science and Technology)

2. Lecturer: 秦晓晴 • Email: qxq@mail.hust.edu.cn • Public email: englishgrad@126.com • Password: 123456

3. An Introduction to this Course • Definitions & classifications of research, and research procedures • The concept of variables • Quantitative research design • Questionnaire survey • Item analysis,validity & reliability • Data exploring

4. Descriptive statistics • Inferential statistics • Writing a research proposal • Reporting a research • Research evaluation

5. Nature of This Course • To give lectures • To hold discussions and presentations • To do self-study • To administer a test

6. Course books and references (1) • Mackey, A., & Gass, S. M. (2005). Second Language Research : Methodology and Design. Mahwah, NJ: Lawrence Erlbaum. • Brown, J. D. (2001). Using Surveys in Language Programs. Cambridge: Cambridge University Press. • Dörnyei, Z. (2003). Questionnaires in Second Language Research: Construction, Administration and Processing. Mahwah, N.J.: Lawrence Erlbaum Associates.

7. Course books and references (2) • Hatch, E. M., & Lazaraton, A. (1991). The Research Manual: Design and Statistics for Applied Linguistics. New York, NY: Newbury House Publishers. • Seliger, H. W. & E. Shohamy, 1999. Second Language Research Methods. Shanghai:上海外语教育出版社. • 刘润清，1999，《外语教学中的科研方法》。北京：外语教学与研究出版社。 • 秦晓晴，2003，《外语教学研究中的定量数据分析》。武汉：华中科技大学出版社。 • 文秋芳，2004,《应用语言学研究方法与论文写作》。北京：外语教学与研究出版社。

8. Definition of Research 1.Dictionary definitions • (1) According to Longman Dictionary, research means advanced study of a subject, so as to learn new facts on scientific laws. • (2) Collin’s Dictionary says research is a detailed study of a subject or an aspect of a subject.

9. 2. Formal definition of research Research is a systematic approach to finding answers to questions. • question—appropriate • approach—systematic • answer—valid

10. Purposes of research • (a) increasing knowledge within the discipline of applied linguistics, and • (b) increasing knowledge within oneself to understand new developments within the discipline.

11. How to increase knowledge • supports the theoretical basis of the discipline • providing evidence for the efficacy of a curriculum, EFL teaching and learning, and so on. • Developing Research Tools. • For most students, the ability to understand research in applied linguistics may be more important than making a research contribution to the profession.

12. Definition of theory Dictionary: A theory is a formal idea or set of ideas that is intended to explain something. Kerlinger (1986: 9): A theory "is a set of interrelated constructs (concepts), definitions, and propositions that present a systematic view of phenomena by specifying relations among variables, with the purpose of explaining and predicting the phenomena" .

13. Types of research Adapted from Brown 2001:1

14. Characteristics of statstical research • Systematic (systematically structured with definite procedural rules) • Logical (based on a step-by-step logical pattern) • Tangible (based tangible, quantifiable information, ie data) • Replicable (able to repeat the research) • Reductive (help form patterns in the seeming confusion of facts) (J. D. Brown 1988)

15. Steps of quantitative research • (1) Ask or form appropriate questions • (2) Review the literature • (3) State hypothesis null (H0) alternative (H1): positive, negative • (4) Select an approach and do the research • (5) Analyze the data • (6) Report results

16. Sources of Research Questions Step 1: Formulating a general research question • Interest and personal experience • Journals of our field • Supervising faculty • Sources outside second language

17. Step 2: Narrowing down or focusing the question • Importance (generate new information or confirm old information in new ways) • Feasibility • List the possible factors comprising the general question

18. Eg. How well do L2 learners perform speech act?(Broad question) • Which speech acts do you want to investigate? • What types of L2 learners are you going to choose? • In what kinds of situations should the speech events be investigated? • Is the research meant to support test development? Material development? Theory development

19. Narrowed Question What is the Chinese EFL learners’ ability to recognize complaint behavior appropriate in an academic university setting? • Total performance –recognition of one portion of the total • L2 learners – Chinese EFL learners • Speech acts – complaint (subcategory) • The overall setting – university setting

20. Activity 1 • Formulate a research question you are interested in

21. Activity 2 Evaluate the following question: • What are the characteristics of the good language learner?

22. From theoretical definition to operational definition • Any terms must be operationally defined • An operational definition is a clear statement of how you judged or identified a term in your research • Good operational definitions can often be drawn from the existing literature

23. Criterion for a good research question The question should • Interest us • Promise new information or confirm old information in new ways (contribution) • Have reasonable scope (feasibility) • Have key terms that are clearly defined and operationalized

24. Factors influencing feasibility • The amount of time and cost needed • Researcher’s prerequisite knowledge • Consistent definition of concepts and terms • Access to data

25. The purposes of research questions • Descriptive • Exploratory • Explanatory

26. Step 3: Review the literature on the topic as completely as possible • Which questions have been answered satisfactorily? • Which seem trivial? • Which are too complex to tackle at the moment? • Which procedures have been used in answering the questions? • What results have been obtained?

27. Step 4: Formulating the research question or the hypothesis • A hypothesis is a statement of possible outcome of research. • Null hypothesis and alternative hypotheses

28. H1andH2 • There is a positive relationship between similarity of relative clause structures in the learners’ L1 and L2 and their performance on L2 relative clauses. • There is a negative relationship between similarity of relative clause structures in the learners’ L1 and L2 and their performance on L2 relative clauses.

29. Ho • There is neither positive nor negative relationship between similarity of relative clause structures in the learners’ L1 and L2 and their performance on L2 relative clauses. • In most research reports, the null hypothesis, even if not formally stated, is tested rather than a directional alternative hypothesis. Why?

30. Activity 3 • Research topic: relationship of language proficiency and spelling test scores • How to state and interpret the null and alternative directional hypotheses • Null hypothesis: • Alternative hypothesis: • Directional, positive hypothesis: • Directional, negative hypothesis:

31. Interpretation of hypotheses • Null hypothesis: There is no relationship between l. proficiency and spelling scores. • The spelling scores will not vary in relation to the l. proficiency of the Ss.

32. Alternative hypothesis: There is relationship … • The spelling scores will vary in relation to the l. proficiency of the Ss.

33. Directional, positive hypothesis: There is a positive relationship between l. proficiency and spelling scores. • Students at higher proficiency levels will obtain higher spelling scores. Students at lower proficiency levels will obtain lower spelling scores.

34. Directional, negative hypothesis: There is a negative relationship between l. proficiency and spelling scores. • Students at lower proficiency levels will obtain higher spelling scores while those with higher levels of l. proficiency will score lower.

35. Activity 4 • Research topic: gender differences in spelling test scores • state a separate null hypothesis for the effect of gender on spelling scores (ignore proficiency) • State all the interaction hypotheses (adding in proficiency and L1 membership) • Ho for gender: • Ho for gender and L1: • Ho for gender and language proficiency: • Ho for gender, L1, and language proficiency:

36. Interpretation • Ho for gender: There is no effect of gender on spelling scores. • Ho for gender and L1: There is no interaction of gender and L1 membership on spelling scores.

37. Ho for gender and language proficiency: There is no interaction of gender and language proficiency on spelling scores. • Ho for gender, L1, and language proficiency: There is no interaction of gender, language background, and language proficiency on spelling scores.

38. Ho for genderrejected: males and females would show different spelling scores. • Ho for gender and L1 rejected: there might be a pattern where for females the scores wouldn’t change across the L1 groups while for men they would.

39. Ho for gender and language proficiency rejected: there might be a pattern where for females the scores wouldn’t change across the proficiency levels while for men they would. • Ho for gender, L1, and language proficiency rejected: it might mean, eg, that females of certain L1 groups who are at certain proficiency levels perform differently than others.

40. Answering the question: Internal and external validity • Internal validity is the extent to which the outcome is a function of the factor you have selected rather than other factors you haven’t controlled. • External validity is the extent that the outcome would apply to other similar situations in the real world.

41. Factors influencing the internal validity • Subject selection (size variability, sample size) • Maturation (developmental changes in subjects) • History (no control of input amount beyond the experiment) • Task direction (no clear instructions) • Attrition • Test effect (the influence of pretest on posttest)

42. Subject selection • In order for research to be generalizable, the sample used should be representative of the general population. Methods used to achieve this representativeness include both random sampling and the sample size large enough.

43. Example of subject selection • 2 groups of learners have been randomly assigned. Each group contains 10 subjects. After the instructional treatment, a test is given to both groups. The average score for group A is 9.3 while 11.3 for group B. The standard deviations for both are 4.15 and 1.56 respectively. Can we say the method used with group B is superior to that with group A? Why or why not?

44. With this difference between the groups for the SD, we would have to question whether test differences might be due to some other, unidentified factor for the score seem to indicate that the 2 groups are drawn from different populations. • The problem for this research is that the subjects are not from a homogeneous population.

45. Maturation • Maturation refers to the general developmental changes in subjects during the course of the research esp. in longitudinal studies. The differences may be caused by the treatment , the maturation in cognitive development and short-term memory capability, or a combination of both.

46. History • History refers to the possible negative effects of the passage of time on the study. For example, learners may have more access to sources of input or use English more outside of class. In this case, we can’t say the difference in performance is caused by the experimental treatment.

47. Task directions • When the instructions are clear, the results may be valid. For example, when a researcher used an instrument of English, the subjects may not understand the instruments accurately.

48. Attrition • By attribution we mean that the composition of the population studied may change the longer the study continues. Subjects may lose interest when data collection is made.

49. Test effect • If a research begins with a pretest, that test can affect performance during the treatment and on future tests. The test alerts subjects as to what teachers expect them to learn. Then they may pay much attention to it during the course of the research.

50. external validity • If a study has no internal validity, then it cannot have external validity. We cannot generalize from the data. • When generalizing from the data set, we are both concerned with IV but EV as well, namely how representative the data are for the group(s) to which we hope to generalize.