1. Learning from Obama: Redesigning Analytics

2. The fundraising challenge • In 2008, Obama campaign raised $750 million • Would not be enough in 2012 $750 million? Not impressed.

3. The fundraising challenge • But fundraising was proving more difficult in 2012 than in 2008 • President less available for fundraising events • In early campaign, we saw average online donation was half of what it had been in 2008 • We had to be smarter, and more innovative

4. Overview • A/B testing in Obama’s digital department • Lessons learned • Don’t trust your gut • Foster a culture of testing • Make it personal

5. Winning with A/B Testing

6. Example: Draft language

7. What impact can testing have? Test sends Full send (in millions) • $2.2 million additional revenue from sending best draft vs. worst, or $1.5 million additional from sending best vs. average

8. Test every element • After testing drafts and subject lines, we would split the remaining list and run additional tests • Example: Unsubscribe language

9. No, really. Test every element. • We also were always running tests in the background via personalized content

10. Then, keep testing • Example: how much email should we send? +6 emails per week

11. The results • Campaign raised over one billion dollars • Raised over half a billion dollars online • Over 4 million Americans donated • Recruited tens of thousands of volunteers, publicized thousands of events and rallies • Did I mention raising >$500 million online? • Conservatively, testing probably resulted in ~$200 million in additional revenue

12. Lessons

13. Lesson #1 Don’t Trust Your Gut

14. Don’t trust your gut • We don’t have all the answers • Conventional wisdom is often wrong • Long-held best practices are often wrong • You are not your audience • There was this thing called the Email Derby… • If even the experts are bad at predicting a winning message, it shows just how important testing is.

15. Experiments: Ugly vs. Pretty • We tried making our emails prettier • That failed • So we asked: what about ugly? • Ugly yellow highlighting got us better results

16. Lesson #2 Foster a culture of testing

17. The culture of testing • Check your ego at the door • Use every opportunity to test something • Compare against yourself, not against your competitors or “the industry” • Are you doing better this month than last month? • Are you doing better than you would have otherwise?

18. When in doubt, test • In a culture of testing, all questions are answered empirically • Example: With the ugly yellow highlighting, we worried about the novelty factor • Maybe highlighting would only work for a short time before people started ignoring it (or being irritated by it). • We decided to do a multi-stage test across three consecutive emails

19. The ugly highlighting experiment • Experimental design: • Determined through this test that novelty was indeed a factor Second Email Third Email First Email Group 1 Group 1 Group 1 Group 2 Group 2 Group 2 Group 3 Group 3 Group 3 Group 4 Group 4 Group 4 Group 5 Group 5 Group 5 Group 6 Group 6 Group 6 Group 7 Group 7 Group 7 Group 8 Group 8 Group 8

20. Lesson #3 Use data to make the user experience more personal

21. Big data ≠ big brother • Testing allows you to listen to your user base • Let them tell you what they like • Whether through A/B testing or behavioral segmentation, optimization gives them a better experience • Usually, the interactions that are the most human are the ones that win

22. Be human! • In general, we founds shorter, less formal emails and subject lines did best. • Classic example: “Hey” • When we dropped a mild curse word into a subject line, it usually won • “Hell yes, I like Obamacare” • “Let’s win the damn election” • “Pretty damn cool”

23. Good segmentation: behavioral • Behavioral segmentation was much more effective than demographic segmentation • Donor vs. non-donor • High-dollar vs. low-dollar • Volunteer status • What issues do people say they care about? • After using A/B tests to create a winning message, we could tweak it slightly for various behavioral groups and get better results

24. Experiments: Personalization • Adding “drop-in sentences” that reference people’s past behavior can increase conversion rates • Example: asking recent donors for more money • Added sentence significantly raised donation rate • Confirmed in several similar experiments …it's going to take a lot more of us to match them. Will you donate $25 or more today? …it's going to take a lot more of us to match them.You stepped up recently to help out -- thank you. We all need to dig a little deeper if we're going to win, so I'm asking you to pitch in again. Will you donate $25 or more today?

25. Conclusions

26. Conclusions • Test everything, especially your gut instinct • Foster a culture of testing • Use data to make it personal