The Logic of Statistical Significance & Making Statistical Decisions

1. The Logic of Statistical Significance & Making Statistical Decisions Textbook Chapter 11 & more

2. Last Day– using Normal Distribution, standard deviation & z-scores to calculate probabilities that what we observe in the sample falls outside a certain range due to sampling error Statistical Significance & Probability

3. Elements in logic of statistical significance (p. 273) • Assumptions regarding independence of two variable • Assumptions regarding representativeness of samples drawn using probability sampling techniques • Observed distribution of sample in terms of two variable • Example: Study of a population of 256 people, half male & half female about confort using Internet to make purchases

4. Expected outcomes if Null hypothesis holds (no relationship between sex & comfort)

5. Illustration of a representative sample for previous case

6. Illustration of an unrepresentative sample for previous case (if null hypothesis holds)

7. But what if there IS a relationship between sex & comfort in the population?

8. Logical foundations of tests of statistical significance • How to explain discrepancy between assumptions about • independence of variables in a observed distribution of sample elements • Observed distribution of sample elements • 1-sample unrepresentative • 2-variables NOT independent

9. Situation in the “real world” Type I (alpha) & Type II (beta) Errors

10. Type 1 & Type II Errors (text p. 278)

11. Use of Type I & II Errors • Establishing error risk levels • Conventional .05 level for rejecting null hypothesis • But exceptions (familywise error) • Avoiding Type II errors • Using interval or ratio-levels of measurement (parametric statistics) • Directional hypotheses (one-tailed tests) • Increase sample size (depends on effect size)

12. Effect Size • Degree to which your variables are interdependent in the population • Could be large or small • Variables can be related but with smaller level of significance (.001 as opposed to .05) • Not the same as the strength of the interdependence (small differences can be important)

13. First step: Decide on Error Risk & Sample size • Some researchers have proposed ideal sample sizes for different expectations about effect sizes (Cohen)

14. Second Step: Selecting appropriate statistical test • depends on research methodology • Parametric statistics assume • Probabilistic sampling techniques • Independence of observations (selection one element won’t influence likelihood of selecting another) • Normal distribution of population • Comparison groups reflect population with same variance • Dependent variables measured at interval or ratios level • non-parametric statistics • Usually presume independence but not normal distribution of variables • Nominal or ordinal levels of measurement

15. Measures of Association and Difference • Depends on research questions & operationalization • Measures of association (ex. Correlation coefficients like Pearson’s r) • Measures of difference (ex. Median split procedure)

16. Steps 3 & 4: Computing Test Statistic & Using Tables of Critical Values • Tables of Critical Values • Numerical guides to sampling distribution of statistic • Indicate likelihood of observations being due to sampling error

17. Deciding to reject or not reject null hypothesis • If statistical value based on sample observations is at least as large as critical value—null hypothesis can be rejected • 5% chance that the null hypothesis might be correct (type 1 error) • If computed value smaller than critical value of table then null hypothesis cannot be rejected (but risk of Type II error)

18. Examples of Five Common Statistical Measures in Textbook • Non-parametric measure of difference (chi-square) • Parametric test of difference between 2 groups (t-test) • Parametric test of difference betwee 3 or more groups that vary on one independent variable (ANOVA) • Factorial ANOVA • Parametric measure of association (Pearson correlation)

19. Example: Chi-Square • Goodness of Fit (One-Variable) • Independence (Two-Variable)

20. Chi-Square Test • Evaluates whether observed frequencies for a qualitative variable (or variables) are adequately described by hypothesized or expected frequencies. • Qualitative (or categorical) data is a set of observations where any single observation is a word or code that represents a class or category.

21. Goodness of Fit • One-Way Chi-Square • Asks whether the relative frequencies observed in the categories of a sample frequency distribution are in agreement with the relative frequencies hypothesized to be true in the population.

22. Goodness of Fit

23. Goodness of Fit • Observed Frequency • The obtained frequency for each category.

24. Goodness of Fit • State the research hypothesis. • Is the rat’s behavior random? • State the statistical hypotheses.

25. Goodness of Fit: Expected Outcomes (with Null Hypothesis) .25 .25 .25 .25 If picked by chance.

26. Goodness of Fit • Expected Frequency • The hypothesized frequency for each distribution, given the null hypothesis is true. • Expected proportion multiplied by number of observations.

27. Goodness of Fit • Set the decision rule. • Degrees of Freedom • Number of Categories -1

28. Goodness of Fit • Set the decision rule.

29. Goodness of Fit • Calculate the test statistic. • Subtract the expected frequency from the observed frequency in each cell. • Square this difference for each cell. • Divide each squared difference by the expected frequency of that cell. • Add together the results for all the cells in the table.

30. Goodness of Fit • Calculate the test statistic.

31. Goodness of Fit • Decide if your result is significant. • Reject H0, 9.25>7.81 • Interpret your results. • The rat’s behavior was not random.

32. Goodness of Fit: An Example • You may have heard, “Stay with your first answer on a multiple-choice test.” Is changing answers more likely to be helpful or harmful? To examine this, Best (1979) studied the responses of 261 students in an introductory psychology course. He recorded the number of right-to-wrong, wrong-to-right, and wrong-to-wrong answer changes for each student. More wrong-to-right changes than right-to-wrong changes were made by 195 of the students, who were thus “helped” by changing answers; 27 students made more right-to-wrong changes than wrong-to-right changes and thus hurt themselves. Using a .05 level of significance, test the hypothesis that the proportions of right-to-wrong and wrong-to-right changes are equal.

33. Goodness of Fit • State the research hypothesis. • Should you stay with your first answer on a multiple-choice test? • State the statistical hypotheses.

34. Goodness of Fit • Observed Frequency • The obtained frequency for each category. right-to-wrongwrong-to-right Observed 27 195 • Expected Frequency • The hypothesized frequency for each distribution, given the null hypothesis is true. • Expected proportion multiplied by number of observations. right-to-wrongwrong-to-right Expected .5*222 = 111 .5*222 = 111 Note: Total number of observations, not 100.

35. Goodness of Fit • Set the decision rule.

36. Goodness of Fit • Calculate the test statistic. right-to-wrongwrong-to-right Observed 27 195 Expected 111 111

37. Goodness of Fit • Decide if your result is significant. • Reject H0, 127.14>3.84 • Interpret your results. • The proportion of right-to-wrong changes and wrong-to-right changes is not equal.

38. Chi-Square • Cannot be negative because all discrepancies are squared. • Will be zero only in the unusual event that each observed frequency exactly equals the corresponding expected frequency. • Other things being equal, the larger the discrepancy between the expected frequencies and their corresponding observed frequencies, the larger the observed value of chi-square. • It is not the size of the discrepancy alone that accounts for a contribution to the value of chi-square, but the size of the discrepancy relative to the magnitude of the expected frequency. • The value of chi-square depends on the number of discrepancies involved in its calculation.

39. Chi-Square Test for Independence (Two-Way Chi-Square) • Asks whether observed frequencies reflect the independence of two qualitative variables. • Compares the actual observed frequencies of some phenomenon (in our sample) with the frequencies we would expect if there were no relationship at all between the two variables in the larger (sampled) population. • Two variables are independent if knowledge of the value of one variable provides no information about the value of another variable.

40. Chi-Square Test for Independence: Sex & Auto Choice

41. Chi-Square Test for Independence • Recent studies have found that most teens are knowledgeable about AIDS, yet many continue to practice high-risk sexual behaviors. King and Anderson (1993) asked young people the following question: “If you could have sexual relations with any and all partners of your choosing, as often as you wished, for the next 2 (or 10) years, but at the end of that time period you would die of AIDS, would you make this choice?” A five-point Likert scale was used to assess the subjects’ responses. For the following data, the responses “probably no,” “unsure,” “probably yes”, and “definitely yes” were pooled into the category “other.” Using the .05 level of significance, test for independence. Definitely No Other Males 451 165 Females 509 118

42. Chi-Square Test for Independence • State the research hypothesis. • Is willingness to participate in unprotected sex independent of gender? • State the statistical hypothesis.

43. Chi-Square Test for Independence • To find expected values: • Find column, row, and overall totals. Definitely No Other Total Males 451 165 616 Females 509 118 627 Total 960 283 1243

44. Chi-Square Test for Independence • To find expected values: Definitely No Other Total Males 451 (475.75) 165 616 Females 509 118 627 Total 960 283 1243

45. Chi-Square Test for Independence • To find expected values: Definitely No Other Total Males 451 (475.75) 165 616 Females 509 (484.25) 118 627 Total 960 283 1243

46. Chi-Square Test for Independence • To find expected values: Definitely No Other Total Males 451 (475.75) 165 (140.25) 616 Females 509 (484.25) 118 627 Total 960 2831243

47. Chi-Square Test for Independence • To find expected values: Definitely No Other Total Males 451 (475.75) 165 (140.25) 616 Females 509 (484.25) 118 (142.75) 627 Total 960 2831243

48. Chi-Square Test for Independence • Set the decision rule. • Degrees of Freedom • (number of columns - 1) (number of rows -1) • (c-1)(r-1) Definitely No Other Males 451 165 Females 509 118

49. Chi-Square Test for Independence • Set the decision rule.

50. Chi-Square Test for Independence • Calculate the test statistic. Definitely No Other Total Males 451 (475.75) 165 (140.25) 616 Females 509 (484.25) 118 (142.75) 627 Total 960 283 1243