How Not to Test Communication Volume

January 30, 2018      Admin

I’m a fan of M+R.  Their free nonprofit tool shed is great for quick calculations for those who, like me, can’t do chi-square or t-tests, in their heads.  Their yearly benchmarks are a must read every year.  (And they are adding retention to it, which is a great addition.)  They are also more open about sharing test results than most, knowing that we rise and fall as a community.  So, yep, a fan.

So I was surprised when during a conference call I received their newsletter and also had it forwarded to me by several alert Agitator readers. While on the conference call I could only see that it was being forwarded in alerts without reading them.  But based on the volume, I fully expected to see M+R on the news fleeing the LAPD in a white Bronco.

But no.  Instead it was a piece defending volume –  emailing your donors more – as a retention strategy.  Agitator readers thought Roger and I may have a viewpoint on it (to put it mildly).

Well, I thought, M+R usually has good data – let’s learn more.  Here’s the evidence they cited:

“Looking at two large organizations that have mature digital fundraising programs:

  • Organization A – sends more than 4x as many emails than the benchmark for large organizations, 6x more appeals, and has an online-only retention rate of 50%, with new donor retention rate above 40%. This doesn’t even factor in the bump in retention all organizations see from adding in multichannel gifts. Those retention rates are above the average for the industry, and above the average for peer organizations that we work with.

 

  • Organization B – sends far fewer emails than Organization A, about the same number as other large organizations, and about 1.4x as many appeals, and has an online-only retention rate of 26%, with a new donor retention rate at 20%. Sending far fewer emails led them to a retention rate roughly half of what Organization A experienced.

Of course, other factors likely play into these retention figures, including mission, brand awareness, and other issues. But at a minimum, this data calls into question whether or not the conventional wisdom that says “bombarding” your list with appeals will drive down your donor retention.”

Let’s start with the truest line of this piece: “other factors likely play into these retention figures, including mission, brand awareness, and other issues.”

To this list I would add Org A may have:

  • A better donation form
  • More customization within each email
  • A better sense of why different donors donate
  • Different offline programs
  • Retargeting ads that increase donations
  • Different acquisition strategies and sources
  • Different acquisition volumes
  • Different donor volumes
  • Greater customization of who gets which email (it’s unclear whether the quantity of email discussed is number of emails sent or number of times the organization sends emails)
  • Greater salience in the news, given the year we’ve just had.
  • Disaster fundraising versus non-disaster fundraising
  • Different size, sector, international v national, etc, etc.

In short, there are hundreds of variables that make Organization A different from Organization B – that’s why they must use different letters in their names.

I know an organization that will send about 15 emails in 2018, three of them asks, and get 80-100% retention rates.  It’s because they just started up and had 10 donors in 2017, most of whom I know personally.  Yet I would not recommend going down to a monthly newsletter, plus end-of-year asks as a strategy.

At least M+R put some data behind their statement, which is better than those who would assert there is a platonic ideal number of communications backed by suspiciously little data. And knowing M+R, they didn’t cherry pick these organizations. But even given all this, there are still far too many variables to control.

The way to test volume of communications is to do it within an organization.  Ideally, you set up a control and a pilot group, running different quantities of communications simultaneously.  If this is not a possibility, you test before and after with the same organization.  There, you do have confounding variables of season or year, but at least you have similar messaging, brand, donation forms, etc.

Tomorrow, we’ll talk about what happens when you actually test this way using a number of case studies.  (Hint: volume as a strategy does not come out looking good.)

But let’s take these M+R data at face value for the moment and look at what they are saying.  They are saying if you quadruple your email volume, you can double your donations.  That means that the first quarter of Organization A’s emails are responsible for the first half of revenue and the next three-quarters of Organization A’s emails are responsible for the second half of revenue.

This is the very definition of diminishing marginal returns.  And you can see it in the M+R Benchmarking data.  In it, they find that that fundraising emails increased by 24% from 2015 to 2016.  However, the click-through rate on fundraising emails went down 12% — nonprofits had to run faster and faster to (largely) stay in the same place.

But, you might say, click-through rates (CTRs) are down across all emails — it’s part of the game now.  If that’s the case, then CTRs should be down about equally across all types of emails.  On the other hand, if the number of emails sent is (at least partly) to blame for lower CTRs, you should see the types of emails that had the greatest increases in sends have the greatest decrease in CTRs.  Sure enough, that’s exactly what the data show.

And I bet you will find it with your own email program as well. If you’ve been increasing the number of emails you send, have you seen a corresponding decrease in click-through rate?

That’s the hamster wheel of diminishing marginal returns. And it’s the central flaw of this volume study (that we will return to later in the week): it assumes that volume is the only lever to be pulled.

What would happen if instead of sending four times the email, Organization A put that time and treasure into:

  • Sending better emails
  • Better targeting emails
  • Discovering donor identities and acting on them
  • Increasing their re-targeting spend
  • Increasing donor feedback

All of these have been shown to increase retention.

Best of all, they are scalable.  We’ve heard the reducto ad absurdum (which, to my disappointment, is not a Harry Potter spell) “well, if you want to ask less, what about if you ask not at all” many a time.  Obviously, there is a sweet spot for every donor (not every organization – what’s best for you globally may not be best for the donor).

But, turning that around, how much asking is too much?  After seeing that adding three more emails to a campaign gets only the same results that the original one did (and that’s what these M+R data purport to show), what is the answer to doubling revenue again?

Because if it’s 16 times the email of the average organization, we all know that’s not tenable.

As Seth Godin put it: “The opposite of ‘more’. It’s not ‘less’.  If we care enough, the opposite of more is better.”

Nick

One response to “How Not to Test Communication Volume”

  1. Chuck Sheketof says:

    Knowing that about 1/2 of our donors either don’t receive emails or don’t click on links, I think another factor is how often you send printed materials and the type of printed materials to everyone – those getting and clicking on emails and those not doing so. It is all about having a conversation and building relationships, not about just email numbers