Meta-Analysis

1. Meta-Analysis

2. HW 8

3. Roulette As I described the rules of roulette in the homework: You place a bet on either black or red. You have a 50% chance of winning if you picked black, and a 50% chance of winning if you picked red.

4. Roulette There are actually lots of different types of bets you can make, but this class is about critical thinking, not gambling or mathematics.

5. PuntoBanco In August of this year, Phil Ivey was playing PuntoBanco and won the equivalent of about $87 million dollars. Like roulette, PuntoBanco is completely a game of chance: there is no skill involved, and your odds of winning are almost 50-50 (though of course the casino has an advantage).

6. “Evidence” = Winning The casino refused to pay Ivey, presumably because it believes he cheated, although the casino has said that an investigation revealed no evidence of cheating. It looks like the only “evidence” against him is that it is extremely unlikely that anyone could win that much money betting on coin flips.

7. Are You a Cheater? For example (and here’s the homework problem), suppose you play roulette (my rules) ten times in a row, and win all ten times. What are the chances of that happening? This is going to be (½)^10 = 1 in 1,024. So, pretty low. If you played once a day, 10 games a day, you would win all your games on average about once every 2.8 years. Are you a cheater?

8. How Many Winners? But that’s just one person playing 10 games a day. There are lots of people who play every day. Suppose 1,024,000 have played 10 games in a row at that casino, and nobody ever plays more or less games (this assumption just makes the math easier). How many people do we expect won all 10 games in a row?

9. How Many Legitimate Winners? Clearly it’s going to be. Chances of Winning x Number of Players = (1/1,024) x 1,024,000 = 1,000. So we expect that there have been a thousand legitimate winners.

10. How Many Cheaters Won? And how many cheaters? I asked you to assume that there are 250 cheaters in every 1,024,000 players. Since the casino has only had 1,024,000 players, we expect 250 cheaters. Since cheaters win 100% of the time, 250 of those 250 cheaters will be winners.

11. Answer to the Homework So there are 1,000 legitimate winners and 250 cheaters who are winners. That means the total number of winners is 1,250. The chances that a randomly selected winner will be a cheater are 250 in 1,250 or 20%. So there is only a 20% chance that you are a cheater!

12. More Realistic Assumptions There are probably fewer cheaters; you can win without winning every game in a row, so lots of big winners likely lose many games; even cheaters probably lose lots of games; people play a lot more than 10 times in a row… etc. But the general idea is the same. If cheating is rare enough, winning is not a good sign of cheating.

13. Error The answer I expected (and just calculated) is 20%. It’s actually the wrong answer, but I will give you credit for it. There was a subtle error in my calculation. I assumed that the number of legitimate winners would be the probability of winning (1/1,024) times the number of total players.

14. Cheaters Can’t Win Fairly! But notice that the cheaters are included in the total number of players, and a cheater will never be a legitimate winner. So the real expected number of legitimate winners is (1/1,024) x (1,024,000 – 250) = 999.75, or ¼ of a person less than 1,000. This obviously doesn’t really matter.

15. The Point People kill themselves when they test positive for HIV. Innocent farmers are kept in Guantanamo Bay for years and tortured because they’ve been turned in as terrorists. People like Lucia de Berk get branded serial killers and put in jail on the basis of circumstantial evidence or DNA tests. People who have fairly won tens of millions of dollars get their winnings denied to them. What do all these cases have in common?

16. The Point In all of these cases, someone has neglected the base rate. They have failed to take into account that When the probability of being an X (HIV positive, serial killer, terrorist, cheater) is low, then even tests with high true positive and low false positive rates are unreliable.

17. Problem #2 It’s a strange fact that when an area of Chicago has more street crime, it has more car crashes, and when it has less street crime, it has fewer car crashes. What could explain this strange correlation? I wanted you to attempt three explanations.

18. Street Crime I had to look up “street crime” because I didn’t know exactly what it meant. Wikipedia quotes London’s police definition: “Robbery, often called 'mugging', and also includes thefts from victims in the street where property is snatched and the victim is not assaulted.”

19. So, Like This…

20. Street Crime Causes Car Accidents There are lots of ways street crime could be responsible for more car accidents. 1. If criminals are running away, they may be reckless crossing the street. 2. If they drive away in their cars, they may drive dangerously. 3. If they buy cars with stolen money, they may care less about their cars. 4. If there are lots of muggings, people may drive more often to avoid the crime that occurs on the streets.

21. Another Reason One of the commenters on the blog post I linked to actually had another theory: 5. Street crime results in more people getting shot (if you rob someone with a gun and they don’t give you their money, you shoot them). This means more ambulances on the roads, which tend to drive fast and dangerously.

22. Car Accidents Cause Streetcrime What about the other causal direction? Can car accidents cause crime? This is less plausible, but there are certainly possibilities. 1. If you need money to fix your damaged car, maybe you turn to crime. 2. If someone you know is killed in a car crash, maybe you get angry at the world and become a criminal.

23. One Additional Possibility I’ve at least heard of crimes that work like this. Someone runs into your car while you’re stopped at a red light or stop sign. You get out to talk with them and they rob you and drive away. So maybe it’s not street crime causes car crashes or car crashes cause street crime, but car crashes = street crime!

24. Common Cause Explanations There are also various common cause explanations, where some variable C causes both car crashes and street crime. This seems reasonable, and that’s the direction that the blog post I linked to considered. What could C be?

25. 1. “Deviant Tendency” According to this explanation, some people have a “deviant tendency,” which is just a fancy way of saying that they like to break the rules, and they aren’t motivated by the same moral concerns as the rest of us. This deviant tendency, so the explanation goes, causes them to break the rules of the road, which brings about accidents, and to break the rules that prohibit robbery.

26. 2. Poverty Not having money, or medical care, or food, or whatever can obviously lead people to become criminals who rob other people. But can poverty cause car crashes? The article suggests two ways in which this might happen: by causing people to be lack empathy, and by causing people to be fatalists.

27. Lack of Empathy Empathy is an emotional connection that we have to other people and animals. It is when other people’s happiness makes us happy and other people’s sadness makes us sad. Some evidence suggests that poverty can lead people to lack empathy, and be unconcerned with the suffering of others.

28. Train Deaths in Mumbai (Bombay) For example, every year in Mumbai, there are six thousand deaths caused by trains. That’s 16 deaths every day in just one city. People die by getting pushed off the platform in front of moving trains, by getting pushed off moving trains, and by getting trampled to death by people crowding onto trains. How can people be more concerned with pushing onto the train than with the people they’re pushing to death?

29. Empathy The suggested explanation is that for so long, the people in Mumbai have been so poor, that they have lost their empathetic connection to others. They don’t feel any sadness or loss when someone else is trampled or pushed off a train. Who cares about that guy? I got what I wanted! (I notice this attitude a lot in Hong Kong too, though obviously people here aren’t so callous as to not care about someone else’s life!)

30. A Bad Explanation? Maybe that’s true, but the article suggests it might not explain the correlation between street crime and car accidents. The reason is that rich people lack empathy too. If you have lots of money, you tend to think of other people as “undeserving peasants” and not care what happens to them.

31. Fatalism Fatalism is the idea that whatever will happen will happen (“que sera sera”). The future is fixed, and there’s nothing that can change it. If poverty causes fatalism, then the poor might think that it’s not worth driving safely or using their turn signals, or whatever, because those things can’t change what will happen.

32. Fatalism This is a better explanation, because rich people tend not to be fatalists. They believe that their hard work and effort is what made them rich, not merely the unchanging hand of fate. The poor are more likely to view their situation as not due to their own laziness or incompetence, but rather due to outside forces beyond their control.

33. Multiple Factors Finally, there’s no reason to believe that the correlation might not result from all of these factors. It might be that sometimes street crime causes car crashes, sometimes car crashes cause street crime, and sometimes poverty causes both car crashes and street crime. When you “add up” all these effects, the correlation between street crime and poverty becomes strong.

34. The Point What is the point of this exercise? People very often argue from a correlation claim to a causation claim. This is typically an inference to the best explanation. What best explains the correlation between chocolate eating and Nobel prizes? Maybe chocolate causes you to win Nobel prizes.

35. The Point As a critical thinker, your job is to evaluate their argument and decide whether to believe the conclusion. To do this, you have to look to see whether there are better explanations than the one being offered. “It’s more likely,” you might say, “that wealth is the common cause here.”

36. Meta-analyses

37. Meta-Analysis A meta-analysis is an analysis of analyses. In clearer terms, it is a study that looks at lots of different experiments that have been conducted on the same problem, and tries to “put together” all of the findings.

38. Motivation & Example Sometimes babies are born prematurely (like me). Unfortunately, premature babies are more likely to suffer and die.

39. Steroids In New Zealand, doctors had the idea that giving steroids to premature babies might improve their chances of survival. They did seven separate studies over nine years. Two studies showed some benefit, but five of the studies were unable to reject the null hypothesis, that steroids did not help. As a result, doctors stopped using the treatment.

40. Blobbogram

41. Blobbogram A blobbogram is a summary of a bunch of studies. Each study is represented by a line, so you can see the seven studies that were conducted on steroids in the previous slide. The line down the middle is the “no effect” line: if an study line crosses it, then that study can’t rule out the null hypothesis, that steroids are no better than placebo treatment.

42. Blobbogram Lines to the left represent positive findings. Since two studies showed positive effects, we can see two lines that are completely to the left of the “no effect” line. The length of a study’s line represents its confidence: longer lines are more uncertain. Even though 5 studies touch the “no effect” line, there seems to be a trend here: the lines tend to be to the left, positive side.

43. Meta-Analysis A meta-analysis is a way of “summing up” all the information contained in different studies of the same thing. The blue diamond to the left of the “no effect” line represents the combined meaning of all the studies: steroids work to save the lives of premature babies.

44. Large Effect Size In fact, the effect size is a reduction in the risk of death between 30% and 50%. Here’s what this means in human terms. In the US, 4 million children are born each year. 12% or about half a million of them are premature. There are about 5,000 neonatal deaths (in the first month of life) due to premature birth and complications associated with it.

45. Probabilities So the likelihood that a premature baby (in the US) will die is around 5,000 in 500,000 or 1 in 100 or 1%. This of course is right now, after we learned to use steroids. So that 1% figure is a 30% to 50% reduction of the earlier figure, which should be 1.4% to 2%.

46. The Human Cost That’s up to twice as many deaths. Between 2,150 and 5,000 babies per year died– in the US alone– because they weren’t given life-saving treatment. This happened for eight years after we had all the information to know that the treatment worked. No one had put that information together in a meta-analysis.

47. Meta-Analysis Now I hope it’s clear why these things are important. Ben Goldacre called the idea of meta-analyses an idea “that has saved the lives of more people than you will ever meet.” And it’s true! But what exactly is a meta-analysis and how does it work?

48. Literature Review The literature review is usually a search of databases containing abstracts of all the published literature or registered trials, like PubMed, Embase, or Web of Science. Researchers look for studies that have been conducted on certain topics like “breast cancer” and “wine”. This is the stage at which a meta-analysis can go really wrong for two reasons: cherry picking and the file drawer problem

49. Cherry Picking If you see cherries in the store, you might notice that most or all of them were ripe and healthy. But you couldn’t conclude that most or all cherries are ripe and healthy. These ones have been selectively picked for good condition.

50. Cherry Picking Cherry picking is another name for “the fallacy of incomplete evidence,” and it’s related to confirmation bias and selection bias. If I want to prove that a treatment works and I pick only those studies that are positive and ignore lots and lots of negative studies, then I’m “cherry picking” my studies.