A/B Testing – can you afford not to? #13NTCab

A/B Testing – can you afford not to? #13NTCab Hugh Dwyer, International Rescue Committee Jacob Colie, Stage Coach Digital Tom Williamson Jackson River

Today’s Agenda • Introductions • What is A/B Testing? • Testing Goals – what should we test? • Testing versus Experimenting • Crafting an A/B Test Plan • Selling your plan • Build, Measure, Learn • Testing Tools • Resources & Downloads • QA A/B Testing

What is A/B Testing? A/B Testing: A/B testing is an experimental approach to web design which aims to identify changes to web pages that increase or maximize an outcome of interest (e.g., donation conversion). As the name implies, two versions (A and B) are compared, which are identical except for one variation that might impact a user's behavior. A/B Testing

Standard Donation Page A/B Testing

A/B Test Donation Page A/B Testing

What is A/B Testing? Multivariate Testing Multivariate testing is similar to A/B testing, but tests more than two different versions at the same time. A/B Testing

Multivariate A/B Testing

Challenges • What are some common challenges to doing A/B Testing? • Resources – technology, time and people • Data sampling size • Is my list big enough? • Are my results significant? • “Do no harm” • Lack of existing test data to build on • Risk vs. Reward • To do A/B testing you need to have an Entrepreneurial Spirit! A/B Testing

A/B Testing Channels • Channel Testing • Email • Advertising • Web • Offline/Direct Mail • Integrated (On and Offline) • Segmentation A/B Testing

Testing Goals – what should we test? Channels and Audience Segments Snail Mail Email Search Advertising New Audiences Leads New Donors Multi-Year Donors Website/Landing Environment A/B Testing

Testing Goals – what should we test? • Metrics and KPIs • Establish benchmark metrics: what are you trying to beat? • Average time on site/page • Open rates • Click-through rates • Form conversion (donations, email signups, etc.) • Average Gift • Cost to acquire • Cost per lead A/B Testing

Testing Goals – what should we test? Evaluating options • Alignment with strategy, context • Examine conversion funnels and donor lifecycle: where are your challenges? • Consider potential returns, & calculate ROI • Where can you move the needle the most? • Quick wins and low hanging fruit A/B Testing

Testing Goals – what should we test? Variables Price points Offer elements Modal windows Page layout Products Design elements Video Usability CTA Multi-step forms Default ask CTA Positioning Ask string Keywords Premiums Images Copy Benefits Subject lines Timing Frequency Communications strategies Testimonials Persuasion assets Content Achievements Expert ratings Security seals Cross-channel Media mentions A/B Testing

Testing Goals – what should we test? • Considerations • Evolutionary vs. revolutionary changes • Frequency and pace of testing • Short-term vs. long-term • Easy road vs. hard road • Embrace (calculated) risks A/B Testing

Testing vs. EXPERIMENTS A/B Testing

When to test? When to experiment? A/B Testing

GOALS Accelerate your REVENUE Provide VALUE for your users A/B Testing

The DIGITAL REVENUE CYCLE A/B Testing

BUILD. MEASURE. LEARN* * Read THE LEAN STARTUP A/B Testing

BUILD an MVP A/B Testing

MEASURE revenue and engagement A/B Testing

LEARN what did and didn’t work A/B Testing

Baseline IDEAL A/B Testing

vs. A/B Testing

short-term vs. LONG-TERM A/B Testing

5% vs. 35% A/B Testing

Examples A/B Testing

MERCY CORPS Retention A/B Testing

IRC Rescue Gifts A/B Testing

Getting started with A/B Testing • KISS – Keep it simple silly • Define metrics to test • Sustainer Acquisition • Average Gift • Gift String • Conversion %, etc. • Identify your top 3-5 items to test, and schedule them over Xmonths! • Decide how you will test each scenario • Control group • Minimum data required to validate • Resources necessary • Internal communications • Implement each test • Build, measure, learn – be patient A/B Testing

Individual Test Cycle A/B Testing

Selling your plan Developing a business case Recommendation Objectives Inputs Analysis Plan A/B Testing

Resources & Downloads • Presentation Materials • PowerPoint - www.jacksonriver.com/nten/ab_testing • Third Party Tools: • Optimizely - www.optimizely.com • Visual Website Optimizer - http://visualwebsiteoptimizer.com • Google Content Experiments • 8 Seconds - http://www.8seconds.net A/B Testing

Questions and Answers • Contact info • Hugh Dwyer • Sr. Director, Information & Operations, International Rescue Committee • hugh.dwyer@rescue.org • Jacob Colie • Senior Strategist & Principal, Stagecoach Digital • jacob@stagecoachdigital.com • @jacobcolie • Tom Williamson • CEO, Jackson River • tom.williamson@jacksonriver.com • @tomwilliamson, @jacksonriver A/B Testing

Evaluate This Session! Enter for a chance to win an NTEN engraved mini iPad! Or, search by session title at www.nten.org/ntc/eval

A/B Testing – can you afford not to? #13NTCab