URBDP 591 A Lecture 10: Causality

URBDP 591 A Lecture 10: Causality Objectives • Internal Validity • Threats to Internal Validity • Causality • Bayesian Networks

Internal validity The extent to which the hypothesized relationship between 2 or more variables is supported by evidence in a given study.

Validity Internal Validity is the approximate truth about inferences regarding cause-effect or causal relationships. External Validity: Assuming that there is a causal relationship in this study between the constructs of the cause and the effect, can we generalize this effect to other persons, places or times? Statistical validity has to do with basing conclusions on proper use of statistics. Construct validity refers to the degree to which inferences can legitimately be made from your study to the theoretical constructs on which those operationalizations were based.

Internal validity

External Validity

Statistical Validity H0 (null hypothesis) true H1 (alternative hypothesis) false H0 (null hypothesis) false H1 (alternative hypothesis) True In Reality We Conclude We accept H0 We reject H1 1-  (e.g., .95) THE CONFIDENCE LEVEL The probability we say there is no relationship when there is not  (e.g., 20) Type II Error The probability we say there is no relationship when there is one We reject H0 We accept H1  (e.g., .05) Type I Error The probability we say there is a relationship when there is not 1-  (e.g., 80) THE POWER The probability we say there is a relationship when there is one

Construct Validity

Threats to the Validity of Research Threats to internal validity Although it is claimed that the independent variable caused change in the dependent variable, the changes in the dependent variable may have actually been caused by a confounding variable. Threats to external validity Although claimed that the results are more general, the observed effects may actually only be found under limited conditions or for specific groups of people. Threats to construct validity Although it is claimed that the measured variables and the experimental manipulations relate to the conceptual variables of interests, they actually may not. Threats to statistical validity Conclusions regarding the research may be incorrect because a Type 1 or Type 2 error was made.

Challenges to Internal Validity a. History b. Maturation c. Experimental mortality d. Instrumentation e. Testing f. Interactions with selection

History Any events that occur during the course of the experiment which might effect outcome. Example: An important event not related to the experiment affects the measurement of pre and post-test.

Maturation Changes in the subjects over the course of the experiment. Example: Age, experience, physical development of participants that leads to increase in knowledge and understanding of the world or behavior which can affect program results.

Experimental Mortality Dropouts from the experiment; especially when the dropouts systematically bias the comparisons Example: If your include pretest subsequent dropouts in the pretest and not in the posttest you will bias the test based on the characteristics of dropouts. And, you won't necessarily solve this problem by comparing pre-post averages for only those who stayed in the study. This subsample would certainly not be representative even of the original entire sample

Instrumentation Any way in which the instrument used for observation or collecting data changes from the pre-test to the post-test. Example: Test, Interview, Measurement Technique or Instrument.

Regression Threat A regression threat, also known as a "regression artifact" or "regression to the mean" is a statistical phenomenon that occurs whenever you have a nonrandom sample from a population and two measures that are imperfectly correlated.

Regression Threat The highest and lowest scorers will regress toward the mean at a higher rate than those who scored close to the mean. There will be a higher degree of regression for unreliable measures than for more reliable ones.

Testing The observations gathered may influence the way subjects behave, thus effecting the outcome. Example: having had the experience of taking the GRE once, without any additional preparation, you are more likely to improve your score on a re-take.

Interactions with Selection When there is a relationship between the treatment and the selection of subjects, this causes a systematic bias which affects causal inference Example: A selection threat is any factor other than the program that leads to posttest differences between groups. These include: history, maturation, test, instrumentation, mortality, and regression.

SIMPSON’S PARADOX Any statistical relationship between two variables may be reversed by including additional factors in the analysis.

SIMPSON’S PARADOXExample by Judea Pearl The classical case demonstrating Simpson's paradox took place in 1975, when UC Berkeley was investigated for sex bias in graduate admission. In this study, overall data showed a higher rate of admission among male applicants, but, broken down by departments, data showed a slight bias in favor of admitting female applicants. The explanation is simple: female applicants tended to apply to more competitive departments than males, and in these departments, the rate of admission was low for both males and females. .

FISHNET (JUDEA PEARL)

Which factors SHOULD be included in the analysis? All conclusions are extremely sensitive to which variables we choose to hold constant when we are comparing, and that is why the adjustment problem is so critical in the analysis of observational studies. According to Judea Pearl, such factors can now be identified by simple graphical means.

Bayes Theorem The basic foundation of probability theory follows from the following intuitive definition of conditional probability. P(A,B) = P(A|B)P(B) In this definition events A and B are simultaneous an have no (explicit) temporal order we can write P(A,B) = P(B,A) = P(B|A)P(A) This leads us to a common form of Bayes Theory, the equation: P(A) = P(B|A)P(B)/P(A|B) (marginalization) which allows us to compute the probability of one event in terms of observations of another and knowledge of joint distributions.

Use of Bayes Rule Often one is interested in particular conditional probability and discovers that The reverse conditional probabilities are more easily obtained. Example: One is interested in the P(disease | symptom), but typically P (symptom | disease) is better known. Causality Example: One is interested in the P(cause | effect), but typically P (effect | cause) is better known.

Bayes Theorem The heart of Bayesian inference lies in the inversion formula which States that the belief we accord to a hypothesis H upon obtaining Evidence e can be comnputed by multiplying our previous belief P(H) by the likelhood P(e H) that e will materialize if H is true. P(H | e) = P(e | H) P(H) / P(e) P(H | e) = posterior probability P(H) = prior probability From the definition of condition probability P(A | B) = P(A,B) / P(B) P(B | A) = P(A,B) / P(A)

Bayesian networks Graphs in probabilistic form • To provide convenient means of espressing substantive assumptions • To facilitate economical representation of joint probability functions, and • To facilitate efficient inferences from observation Bayesian networks provide a language for qualitatively representing the conditional independence properties of a distribution. This allows a natural and compact representation of the distribution, eases knowledge acquisition, and supports effective inference algorithms.

Bayesian networks Bayesian Network is a Directed Acyclic Graph (DAG) in which 1. Each node denotes some random variable X 2. Each node has a conditional probability distribution P(X | parent X) The intuitive meaning of an arc from node X to node Y is that X directly influences Y.

Typical Bayesian Network A Bayesian network representing dependencies among five variables:X1 = season of the year X2 = rain X3 = sprinkler is on X4 = pavement get wet X5 = pavement will be slippery

Causal Relationships and their stability S1: Turning the sprinkler on would not affect the rain S2: Belief in the state of the rain is independent of knowledge in the state of the sprinkler S2: Unless we learn what season it is Given X1, S2 changes from true to false once we know that the pavement is wet S1: Remain true regardless of X1 or X4

CONDITIONAL INDEPENDENCE Two events, A and B, are said to be independent if P(A and B both happening)=P(A happening).P(B happening). The two events do not affect one another. A more general result can be derived for conditional probabilities. Two events, A and B, are said to be conditionally independent if P(A and B both happening | Another event C) = P(A happening | C).P(B happening | C). Where we already have information about the situation (through our knowledge of event C), knowledge of event A will not enable us to change our estimate of the probability of event B.

A B C A B C D B C D A Types of Connections • Serial Connections • Diverging Connections • Converging Connections

D-separation • Two variables A and B are d-separated if for all paths between A and B, there is an intermediate variable V such that either: • The connection is serial or diverging and V is known • The connection is converging and neither V nor any of V’s descendants have received evidence • If A and B are d-separated, then changes in the certainty of A have no impact on the certainty of B

Bayesian network inference Compute the posterior probability distribution for a set of query variables, given values for some evidence variables.

Explaining away R = rain S = sprinkler W = Watson’s grass is wet H = Holmes grass is wet R S W H Explaining Away: More probable explanation given the data can explain away other explanations. When nothing is known, R and S are independent. However when we obtain information about H, R and S are dependant.

URBDP 591 A Lecture 10: Causality