What the Simpsons Can Teach Us About Retention Rate

June 19, 2018      Kevin Schulman, Founder, DonorVoice and DVCanvass

You’ve just made a wise decision: you are investing more in both donor-focused retention efforts and new ways to bring in the right donors for your program.  Surely, your file will grow and your retention rate will increase.

But when you look at the stats a year later, your file has grown.  But your retention rate is down.  Did you invest poorly?

Congratulations.  You’ve just found Simpson’s paradox.

This has nothing to do with Homer, Marge, Lisa, and Bart.  Simpson’s paradox is a statistical oddity where subgroups can go one direction while the whole of the sample goes the other direction.

Let’s take your retention example, simplified with two subgroups and roundish numbers to make the math easier.  In year one, you have:

New donors Multi-year donors Total
Total from previous 100,000 100,000 200,000
Retained 25,000 75,000 100,000
Retention rate 25% 75% 50%

In year two, you have:

New donors Multi-year donors Total
Total from previous 200,000 100,000 300,000
Retained 60,000 80,000 140,000
Retention rate 30% 80% 47%

New donor retention went up.  Multi-year donor retention went up.  Overall retention went down, because the make-up of the file changed.

Hence, no one should ever be given the sole goal of improving overall retention rate.  They can cut new donor acquisition and nothing else; overall retention will increase with no impact on subgroups.

Neither should you only look at subgroup retention rates.  This ignores strategies that intentionally change the balance of your file.  For example, you’ve probably heard that you should spend as much to reactivate a lapsed donor as you would to acquire a new donor from outside your organization.  But look at your lifetime value and retention rates for reactivated donors versus newly acquired donors.

My bet is that the former is worth more than the latter.  Thus, you may want to reallocate your acquisition spend.  If you reactivate more donors, who have higher retention rates, your overall retention will go up even if the subgroup retention rate doesn’t.

So, you want to look at both overall retention rate and those of subgroups.  Some helpful subgroups at which to look:

  • Lifecycle segments, per the discussion above
  • Let’s use direct connection to the cause versus none.  If you know these identities have value for you, you will want to look at retention separately to see if the customizations and segmentations you are doing are helping.  This will also help you refocus your overall retention efforts as you work to acquire more of your better-identitied donors.
  • Medium/channel. That email is going to help or hurt your retention effort more in online giving than in other channels.
  • Commitment level. Higher commitment donors have higher lifetime values and retention rates.  When you ask for and get this information at point of acquisition, as demonstrated here, you can increase your efforts to retain these donors and save their higher values.

Moreover, you can also train your acquisition efforts to find more donors like these committed donors, making sure you are bringing the right people into your organization.

So, Simpson’s paradox shows us we need to be looking at both the forest and the trees.  We need to improve our efforts to increase retention within each subgroup and to also change our donor mix to our overall advantage.

Nick

7 responses to “What the Simpsons Can Teach Us About Retention Rate”

  1. Caity Craver says:
    June 19, 2018 at 9:34 am

    Nick,

    Great post – and timely for year end planning.

    As you point out, this Paradox can deceive us when evaluating annual performance. It can also wreak havoc when reviewing campaign level performance. I’ve cringed through many discussions as Agency and Client incorrectly make assumptions on control / test results because of faulty test design.

    We are all beginning our ‘year end’ planning. During this season we will create more tests than in any other time of the year. A warning: Avoid the Paradox and take a scientific approach to test design. A few simple, yet vital, rules to follow:

    * Control and test groups are characteristically similar
    * Panels or lists are the same quantity
    * Test quantities are statistically significant

    In an industry that LOVES to test, test, test, we all become Homers [and get deceived by the Simpson Paradox] if we don’t follow apply some basic science to our testing plans.

  2. Kimberly Proffitt says:
    June 19, 2018 at 10:15 am

    This is such an important post. So often organizations use only the top-line number to make decisions. In my work we spend a lot of time doing exactly what you mention — going several levels deeper, looking at things from various angles. Why is the overall retention rate what it is? Once we better understand the drivers, we can then make recommendations and decisions that are more strategic and lead to long-term stability.

  3. Jay Love says:
    June 19, 2018 at 11:00 am

    Great post Nick!

    Looking more than one layer deep is so very important…

  4. Gayle L. Gifford, ACFRE says:
    June 19, 2018 at 11:54 am

    Great post! Every so often I’ve heard organizations talk about their amazing 90% retention rates of donors (who aren’t on autopay). Digging just a tiny bit deeper, they have done little acquisition in any recent memory and their donor base just keeps eroding and eroding.

    Analyzing and investing in growing your funding is both simple and complex at the same time. Thank you for taking on these issues.

  5. Pamela Grow says:
    June 19, 2018 at 5:46 pm

    Another fabulous post, Nick. I love that you always make us think and dive deeper. Thank you!

  6. Cindy Courtier says:
    June 20, 2018 at 9:48 am

    Too much deep through this early in the am makes my brain fizz. I’ll come back and read later after coffee.

    But you’re absolutely spot on about statistical significance and testing. “I cringe every time I hear “This won because it got 5 or 7 or 57 more responses…” when the total universe was just to small to support that assumption.

  7. Nick Ellinger, VP of Marketing Strategy, DonorVoice says:
    June 20, 2018 at 12:58 pm

    Thanks for a great discussion. Gayle, another trick I’ve seen for 90% retention rates is a pseudo-measure that people call retention + reactivation: they put lapsed donors in a different bucket and count reactivated donors in the numerator, but not the denominator, of retention. As such, you could get a retention rate over 100%, which is a crime against statistics.

    And, of course, Caity is right on about testing methodology. I would add as a bullet “you have a hypothesis and a reason to believe it”. So much testing is done for the purpose of testing, rather than to support or oppose a hypothesis. If you think something will increase average gift, and it lowers it but boosts response rate, you have to think of why that happens

Ask A Behavioral Scientist

    Behavioral Science Q & A

    The Agitator Tool Box

    Ideas, applications, tools, processes, and case studies of break-through solutions in fundraising, including: