An A/B test feels scientific because it is.  The audience is randomly split, version A beats version B, some runs a stat sig test and voila, everyone gets A from now on.

But what about the people that preferred B?  A 55% to 45% clear win still leaves plenty of people preferring B and getting a diet of A.  That leaves money on the table but that’s not the biggest issue.

A/B tests assume the contact itself was necessary. They ask which email, ad, text or mail package worked better but rarely ask whether sending either one created any new behavior.  That is the question that the wonky “incrementality” word is raising.

Someone clicks an email and donates, the email gets credit. Someone taps the SMS link and gives, SMS gets credit. Someone sees a retargeting ad, returns to the site and converts and the ad platform gets credit.

None of that proves the contact caused the gift.

What if the donor was always going to give at some point?  The extra contact captured a donation that was already coming while adding cost and reducing margin.

A large field experiment with a mattress company shows how badly attribution can mislead. Facebook reported a 7.47 return on retargeting spend, far above the 2.21 return for prospecting. Yet when researchers randomly varied $5.7 million in advertising across nearly 200 geographic markets, prospecting produced significant incremental orders – i.e. orders that would not have happened without the advertising.

Retargeting generally produced no measurable increase in orders or sales. It was excellent at finding people likely to buy and apparently poor at causing more of them to buy.

That new SMS program may generate gifts while adding almost no revenue. The extra mail package may collect gifts that would have arrived through the next package. Email may intercept donations from people who were already headed to the website.

Growth cannot come from endlessly spending against the warmest people simply because they are easiest to convert. Existing donors and highly engaged prospects will always look efficient in attribution reports since they are already more likely to give.

The relevant question is whether the spend changed their behavior.

A/B testing still matters. It can tell you which treatment performs better among the people exposed. Personalization can improve on that by recognizing that different people prefer different treatments.

But neither answers whether the treatment was needed.

That requires holdouts. Suppress part of the audience from the extra mailing. Introduce SMS to one randomized group and withhold it from another. Turn paid media off in selected markets. Then compare total donors, total revenue and total margin across all channels.

Otherwise, we’re optimizing the garnish without checking whether anyone ordered the meal.

Kevin