charity: water Goes Mailing?, Part 2

July 27, 2022      Kevin Schulman, Founder, DonorVoice and DVCanvass

Personalized Matching Findings

Winning execution Openness, losing on Agreeable. 

The holdout group has no hard, incremental costs to service it – it’s all digital.  The net or profit margin for the holdout in the winning Openness treatment is higher than the letter group margin.  But, you got a lot more people “engaged” with the brand again; giving, exposure through different channel, etc. by including the mail in the mix.

So, what about digital, couldn’t this same thing be done there to ‘beat’ the one-size holdout control?  Glad you asked and as usual, charity: water was thinking ahead too.

They ran this email test with the last email in a 2-week, country specific campaign.

The changes are subtle, an extra line of copy in the Openness (middle) and Agreeable (right) treatments versus Control (left).  Plus, copy change tailored to match Personality on the button call out text.

That’s it.  Everything else was identical.

Results?

Mirrors the mail test with Openness doing better than Agreeable but in this digital test they both beat the control albeit only directionally with the latter.

Summary

Test, test, test.  That’s the sector mantra.  There ought to be a predicate, think different, then test, test, test.  A big part of charity: water success comes from thinking differently, it’s built into their DNA.  Not only are they receptive to new ideas, they seek them out.

This testing, like all testing, had constraints.  It also had success and failure.  All of it contributes to the knowledge base and idea factory for charity: water.  That is the real value.  Inherent in this is getting beyond just format and channel as the lens onto the world.  People are different, we can either market and fundraise to these differences or ignore them.  Both are choices and charity: water made theirs.

What will you take from this?

Kevin

P.S.  As promised, here is a link to both charity: water posts as a single case study.

6 responses to “charity: water Goes Mailing?, Part 2”

  1. thanks Kevin, this is great and I’m absolutely for testing, but I can’t quite see how there is a statistical difference between 20, 6 and 2 conversions, they’re all the same.

    Unless you have at least 100 responses, I don’t see how there can be a 95% significant difference.

    As I don’t know how many were mailed for the letter and post card, not sure if those are significant either.

    I can say that if the goal was prospecting of new donors, 1.4% and 1.6% are quite good and my conclusion would be that direct mail could also work for charity:water. And it certainly would help with generating donors who give through both channels.

    Having said this, the tests are great as they are aimed at being more donor centric and testing new things and new wording especially.
    I just read the new study from Adrian Sargeant on the use of certain words, being more donor centric (he calls it donorcentricity 3.0) and it’s fascinating how it can help lead to more … especially if fed by words generated through donor surveys. check it out.
    https://www.philanthropy-institute.org.uk/donor-centricity-30
    cheers, erica

    • Kevin says:

      Hi Erica, thanks as always for reading and commenting. Let me correct the comment about statistical testing for the record and so that the misconception doesn’t get repeated. There is nothing about 100 responses required nor any sample size requirement at all for significance testing. Sample size does matter in that if it’s large enough you could officially flag something as “statistically significant” even if it is practically irrelevant – e.g. a hundredth of decimal point difference in response could be magically flagged as “statistically significant” if the sample sizes were large enough. In this case, it is accurate reporting, there is 95% confidence that the Openness test beat the control. I hope the 100 thing isn’t out there in the ether.

      A few other clarifications, this testing was not for new donors, it was folks who lapsed as sustainers. The Openness direct mail test did beat the holdout group, statistically. The Agreeable direct mail did not, they are statistically tied but the mail has cost attached the holdout condition does not so on a net basis it loses.

      Openness on a net financial basis also “loses” since there is never any direct cost for the holdout treatment that is all digital. However, it did still produce a profit/net (albeit lower margin) and it did get more donors back into the fold.

      There is lots of nuance here. As we stated in the post, there were testing successes and failures but all of it contributes to the knowledge base and idea factory for charity: water.  That is the real value. 

      Kevin

  2. Sarah Robie says:

    Clearly, if charity:water had a personality, it would fit in the Openness category! Thank you so much for sharing these tests and results. I’m curious to know why, with the email test in particular, you tested subtle tweaks between the personalities. Did you want to see the effect of minor changes? Would you use the results as motivation to test bigger changes, such as subject line, headline, image, etc?

    • Kevin Schulman says:

      Sarah, nailing it on the Big Five, yes, it would definitely be high in Openness. Good question on the small tweaks. This email test was charity: water working from the DonorVoice “crib sheet” and having been exposed to all the discussion and rationale on the letter copy – i.e. the process became de-facto training for charity: water. And they own the Personality Tags on their house file for permanent, ongoing usage.

      So, they made these changes and were a bit unsure if they were doing it right and therefore, wanted to make fewer vs. more in case they were off the mark in the tailoring. We gave them high marks on the tailoring and expect they’ll do more testing as the learning and growing continues since we know those Open folks love learning and have a growth and continuous improvement mindset.

  3. Joanne Wilson says:

    Will charity:water be rolling out with a direct mail program based on these results? Or will there be further testing? I’m curious about what their next steps will be!