It was an A/B test winner.  The Bavarian State Opera had a fundraising ask post-online-ticket purchase.  They found if they forced ticket purchasers to state either “I have donated already” or “No, thank you” to bypass this ask, response rate went up one percentage point and average gift increased by $.46.

Increased average gift and response rate?  Well, that’s a winner!  Roll it out, then roll in the Euros, no?

Maybe.  Maybe not.

Because that’s only the first part of this interesting study from Adena and Huck.  The researchers then looked at the next opera season.  What did they find?

“Customers who were forced to admit being a non-donor during an online fundraising buy significantly fewer tickets online in the next season. The online return from those customers is lower as well.  However, when accounting for all means through which tickets can be purchased the results turn non-significant in specifications with the full set of controls. This suggests substitution between different means to buy tickets. Customers avoid the online ask by purchasing tickets on the phone or at the box office.”

In this case, the BSO probably chose to keep the questions, as patrons were able to get their tickets other ways and preserve revenues.  But we might not want to add these options on our own sites, because online donors frequently don’t give other ways.  Thus, the substitution for making an online donation is likely not donating at all.

This has a few implications for your testing and fundraising:

People will avoid the ask if they don’t want it.  Previous research had shown that people are more likely to exit a grocery store on the side where there is no charity ask.  Similarly, people are less likely to be home when they know a door-to-door fundraiser is coming.  If people don’t want to give, they put themselves in a position not to be asked.

This happens for mail donations as well.  People who get more solicitation requests say they write down when they gave to a charity last, try to find out when last they gave before donating, limit their giving to charities of a certain type, and/or put the “maybe” solicitations in a box or pile for later reference.  And, not surprisingly, the more mail they get, the more they agree with the statement “I feel I must protect myself from the mail I get from charities.”

Thus, it’s to our advantage to make our asks when people are in a position to say yes.  Look at this example from the National Committee to Preserve Social Security and Medicare, where those who ask for less mail end up being more valuable on fewer asks.

Satisfaction matters.  Two weeks ago, I did yet another rant about how emailing the heck out of people isn’t a retention solution.  One of the reasons there shows up here: if people feel bad as the result of their interaction with you, they’ll avoid you.  That’s likely why both raw and per donor donations have decreased as we as a sector have increased our email volume.  Annoying your donors and hoping they won’t notice is not a viable strategy.

One-time A/B testing can lead you down the wrong path.  If you tried this with a typical A/B test, you could increasing your one-time donations, but killing your retention.  This shows (one of) the limitations of the A/B test – it doesn’t account for long-term impacts.  This is why we recommend pilot testing large-scale strategic decisions like cadence prototypes, the value of a donor identity, and more.

Of course, donor feedback can be the canary in the coal mine.  Donors, and those who abandon donation forms, won’t be shy about sharing techniques that rub them wrong.  But looking at long-term results is always helpful – perhaps there are some A/B tests in your past where you backed the wrong horse.