andco.uk

1. Myths and Magic Tricks Oliver Mantell www.andco.uk.com

2. What I’m Going to Talk About The two basic myths about data: Myth 1: Statistics are irrelevant Myth 2: Statistics are hostile Types of magic tricks: What you can find out with no data at all Getting complexity from simple data Getting simplicity from complex data

3. Warm-Up and Stretches www.andco.uk.com

4. What are the odds? If your coin is fair, that: It turns up heads on the next toss? It turns up heads more often than tails? It turns up tails, if it turned up heads last time?

5. What are the odds? If your audience are 50:50 male to female, that: The next random person you survey is male? You survey more women than men? The next randomly selected visitor is male, if the last was female?

6. What are the odds? If by knowing that the sample is 50:50 male to female, you can work out the odds of all 10 people you survey being male... ...you can also work out the probability that the sample is 50:50 male to female, if all 10 people you survey are male. You don’t need to be able to work out the exact answer, just to know that it IS possible to work out.

7. Myths www.andco.uk.com

8. MYTH 3: Randomness is difficult Randomness makes things easy: its when they’re not random it gets difficult.

9. MYTH 4: My Survey is Sampling My Visitors Sampling works on selecting items from a population that are equally likely, so that they can represent the whole. So you’re probably sampling visits, not visitors. VS.

10. MYTH 5: 51% is bigger than 50% Just because your result is bigger than before, doesn’t mean that that’s what is happening in reality. You have to look at the margin of error. If it’s different, but not significant, it isn’t different. Most newspaper stories about changes in opinion polls aren’t true.

11. MYTH 6: Significant changes matter There’s a difference between a change being real and it being big. You should only care about substantial AND significant changes.

12. MYTH 7: ‘Significant’ changes are significant We only said we wanted to be 95% sure. If you look at enough examples, one in twenty won’t be significant, although it will look it. LIAR? % % % % % % % % % % % % % % % % % % % %

13. MYTH 8: A Bigger Sample is Always Better The fewer people you ask randomly, the less chance of a significant result. But asking more, non-randomly, doesn’t help.

14. MYTH 9: Changes to outliers show changes to the odds If you praise people who do well, they do worse. If you shout at people who do badly, they do better. That doesn’t mean that praise is bad or shouting is good.

15. MYTH 10: 67% of Our Focus Group Liked It There’s a reason qual and quant are kept separate. What would it take for that result to be totally different? Erm...? Yes! No!

16. Magic tricks www.andco.uk.com

17. Say It Ain’t So, Joe! ‘Null hypotheses’ are a very powerful trick. ‘A must be true, because B happens’. Assume A isn’t true: does B still happen?

18. Welcome to Monte Carlo! It’s ok to completely make up the numbers (sometimes).

19. Slice and Dice What if: 50% of all visitors go to the cafe. and 50% of all visitors are vegetarian. OR

20. Judge by results: segmentation by response Response-based – based on likelihood of an answer. Uses same significance tests mentioned earlier. Automatic, shows you what has the biggest effect (and is significant).

21. Judge by results: segmentation by response Example:

22. Judge by results: segmentation by response Example:

23. Judge by results: segmentation by response

24. Birds of a feather: segmentation by cluster Based on best grouping of clusters.

25. Birds of a feather: segmentation by cluster Shows ‘natural groups’ within your audience Gets beyond using single categories to describe visitors Identifies similarities and differences between individuals based on patterns in whole audience.

26. Birds of a feather: segmentation by cluster Example: Used attitudinal and behavioural only Showed real differences that made sense There were real demographic differences between the groups.

27. Birds of a feather: segmentation by cluster

28. Summary Statistics matter, but they have to be used with care. Used well, however, that can provide real insight that help you to make decisions and do things better.

29. Just To Remind You... Myth 1: Statistics are irrelevant Myth 2: Statistics are hostile Myth 3: Randomness is difficult Myth 4: My survey is sampling my visitors Myth 5: 51% is bigger than 50% Myth 6: Significant changes matter Myth 7: ‘Significant’ changes are significant Myth 8: A bigger sample is always better Myth 9: Changes to outliers show changes to the odds Myth 10: 67% of our focus group liked it.

30. Just To Remind You... Trick 1: Say it ain’t so! – Null hypotheses Myth 2: Welcome to Monte Carlo! – Using simulations Myth 3: Slice and dice – Cross-tabulation Myth 4: Judge by results – segmentation by response Myth 5: Birds of a feather –segmentation by clusters

31. Thank you. Any questions? www.andco.uk.com

32. Contact us Leeds office 46 The Calls LS2 7EY Telephone: 0113 234 6857 Email: info@andco.uk.com www.andco.uk.com